diff --git a/CLAUDE.md b/CLAUDE.md index 31195a0f..afc50399 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,7 @@ python -m dw.test the seam, and the test or check that enforces it. Open it before grepping - it names the module, and that module's docstring holds the detail. -- `dw.serve` runs every job in one persistent spawned worker (`dw/worker.py`, managed by +- `dw.serve` runs every job in a persistent spawned worker (one per `--devices` card) (`dw/worker.py`, managed by `dw/worker_manager.py`) that keeps models cached between runs, so a change to engine code needs a server restart before a job sees it. - The packaged `dw/workflows/` is what a `builtin:` step names (resolved in @@ -71,8 +71,9 @@ The guides in `docs/`, by topic: ### Claude Code plugin `.claude-plugin/marketplace.json` publishes the `dw` plugin in `plugins/dw/`: one -composition skill per model family (`minimax-h3`, `minimax-music3`, `ltx-2.5`) that -chooses a template for a request's shape and states the family's hard rules, plus +composition skill per model family (`minimax-h3`, `minimax-music3`, `ltx-2.5`, +`kandinsky-6`) that chooses a template for a request's shape and states the family's +hard rules, plus cross-cutting composition skills (`script-to-video`, `series-episodes`) - shapes above the families that orchestrate the decision trees and cast consistency across multiple generations. Every skill the directory holds is named in `plugins/dw/README.md` and here, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 07281392..b8593b08 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -30,8 +30,9 @@ one to open. | On-demand and offload | `dw/pipeline_processors/placement.py`: `place_component`, `has_component_group_offload` | On-demand residency and `group_offload` cannot both be set on one component, and either one stops the whole pipeline from being moved to the device. | `tests/test_modular_pipeline.py::TestOnDemandResidency::test_group_offload_and_on_demand_together_are_rejected`, `tests/test_modular_pipeline.py::TestOffloadDevice::test_component_group_offload_is_not_moved_to_device` | | LoRAs | `dw/pipeline_processors/adapters.py`: `active_loras`, `load_loras` | A `loras` entry whose `model_name` is null is switched off. | `tests/test_lora_disable.py::TestLoadLoras::test_an_all_null_entry_loads_nothing` | | Step progress | `dw/pipeline_processors/progress.py`: `reported_progress_bars` | A pipeline with no step callback reports its denoise steps through its progress bars. | `tests/test_modular_progress.py::test_every_advance_of_the_bar_is_reported` | -| Adding a task | `dw/tasks/task.py`: `register_command` | A task is a function registered with `@register_command`, and the signature of its implementation is its argument schema. | `tests/test_task_discovery.py::TestDescribeTask::test_signature_becomes_the_schema` | -| Task argument domains | `dw/task_domains.py`: `check_arguments` | A numeric domain that a signature cannot express is declared in this table, and it is checked at validation and again at run time. | `tests/test_task_domains.py::TestTheStaticPass::test_a_negative_frame_count_is_refused_at_its_path`, `tests/test_task_domains.py::TestAtRunTime::test_slice_audio_refuses_a_negative_count` | +| Community pipelines | `dw/community_pipelines/` (e.g. `pipeline_ltx2_refine.py`: `LTX2RefinePipeline`), `dw/introspection.py`: `community_pipelines`, `load_allowed_class` | A pipeline dw ships is named by its dotted path (`dw.community_pipelines..`), loads untrusted because it is under `dw`, and is listed by `list_pipelines` and resolved by `get_pipeline_signature` - only the modules in that directory, never a caller-chosen import. `LTX2RefinePipeline` VAE-encodes `video` at `width` x `height` and passes it unnormalized as `latents`, because `LTX2Pipeline.prepare_latents` normalizes 5-D latents itself. | `tests/test_ltx2_refine_pipeline.py::TestIntrospection::test_only_the_shipped_modules_resolve`, `tests/test_ltx2_refine_pipeline.py::TestEncode::test_the_latents_are_left_unnormalized_for_the_parent` | +| Adding a task | `dw/tasks/registry.py`: `register_command`, `RegistryTable` | A task is a function registered with `@register_command`, and the signature of its implementation is its argument schema. Its validate-time rules are declared on the same decorator - `domains`, `choices`, `static_check`, `whole_numbers` and `media_arguments` - and `TASK_ARGUMENT_DOMAINS`, `TASK_ARGUMENT_CHOICES`, `TASK_STATIC_CHECKS`, `TASK_WHOLE_NUMBER_ARGUMENTS` and `TASK_MEDIA_ARGUMENTS` are read-only views derived from them, so there is no second table to forget (#692). The registry lives in `dw/tasks/registry.py`; `dw/tasks/task.py` re-exports it, and a task module may register its own handler, as `dw/tasks/beats.py` does, provided `dw/tasks/task.py` imports it (`docs/TASKS.md` *Adding a task*). | `tests/test_task_discovery.py::TestDescribeTask::test_signature_becomes_the_schema`, `tests/test_task_registry_rules.py::test_every_derived_table_is_the_registrys_view`, `tests/test_task_registry_rules.py::test_every_static_check_is_registered_to_exactly_one_command`, `tests/test_task_registry_rules.py::test_every_media_argument_is_a_parameter_of_its_command` | +| Task argument domains | `dw/task_domains.py`: `check_arguments`, `task_argument_errors` | A numeric domain that a signature cannot express is declared on the command's `@register_command(domains=...)`, and it is checked at validation and again at run time. A numeric argument is read by `whole_number` or `real_number`, the only numeric coercion in `dw/tasks/`, and validation applies the same rule (`number_problem`), so a value is refused at both or at neither: `"3.0"` is a whole number, `"3.5"` for one of `whole_numbers` is not, and `true` is no number (#774). `Task.run` coerces every declared argument through `coerce_arguments` before dispatch, so a handler sees a number, and an argument that has no range besides being a number declares `finite`. A command's cross-argument rules are its registered `static_check`, which `task_argument_errors` runs at validation (`attribute_voices`' `voices_argument_errors`, kept in `dw/tasks/voice_attribution.py`, among them). A literal enum choice (`choice_errors`) and the `ingredients_grid` image count against `max_images` (`ingredients_grid_errors`) are refused at validation by the same module. So are `crop_face_track`'s cross-argument rules (`face_track_problems`, `face_track_errors`: crop size a multiple of the step's `multiple` (32 by default), `remainder` below `modulus`, `gate_zero` above `gate_full`, padding and confidence limits), which the task checks again at run time, as `analyze_beats`' are too (`beats_problems`, `beats_errors`: `min_bpm` below `max_bpm`, and anchors that are one kind per list, ascending, with whole beat indexes; `tests/test_analyze_beats.py`), and `plan_cuts`' (`cuts_problems`, `cuts_errors`: a bare-string transcript (`transcript_problem`, which `plan_cuts` also calls at run time), `min_scene_s` above `max_scene_s`, `segment_by: "beat"` without beats), and `select`'s index (`select_index_problem`, `select_rule_problems`: a whole number, at least 0, and below the candidate count when the count is known, i.e. at run time or when `candidates` is a written list), and `apply_lut`'s (`lut_source_problem`, `palette_problem`, `lut_errors`: exactly one of `lut` or `palette`, and a palette of 2..16 `#rrggbb` colours, re-checked at run time by `check_lut_source`), and `check_face_detector_source` vets its Hub repo and `.onnx` file before any download. | `tests/test_task_domains.py::TestTheStaticPass::test_a_negative_frame_count_is_refused_at_its_path`, `tests/test_numeric_coercion.py::TestGradeExposure::test_a_string_number_grades_like_the_number`, `tests/test_task_domains.py::TestAtRunTime::test_slice_audio_refuses_a_negative_count`, `tests/test_ingredients_grid.py::TestStaticValidation::test_bad_layout_and_fit`, `tests/test_face_track.py::TestStaticValidation::test_bad_literal_is_an_error_at_its_argument`, `tests/test_face_track.py::TestCropFaceTrack::test_bad_arguments_are_refused_at_run_time`, `tests/test_face_track.py::TestStaticValidation::test_a_declared_grid_is_accepted`, `tests/test_face_track.py::TestDetectorSource::test_traversing_repo_is_refused`, `tests/test_select_validation.py::TestSelectErrors::test_an_out_of_range_literal_index_names_both_ends`, `tests/test_lut.py::TestPaletteAtValidate::test_both_refuse_naming_the_two`, `tests/test_lut.py::TestPalette::test_a_bad_palette_refuses_naming_the_entry` | | Variables | `dw/variables.py`: `set_variables`, `argument_errors`, `resolve_variable_values` | `argument_errors` folds a caller's arguments in exactly as `set_variables` does when a run starts, so a bad name or value is refused before anything is queued. | `tests/test_variables.py::test_argument_errors_reports_a_dict_passed_for_a_string_variable` | | List variables from the CLI | `dw/run.py`, `dw/variables.py`: `get_value` | A `name=value` string given for a list variable is split on commas, so a list of objects can only be passed as JSON over the API or MCP. | `tests/test_variables.py::test_set_variables_list_default_splits_on_comma` | | `for_each` expansion | `dw/for_each.py`: `expand_for_each` | Each entry becomes an ordinary step named `@`, after substitution and before the reference check. | `tests/test_for_each.py::TestNaming::test_member_name_joins_with_at` | @@ -65,28 +66,37 @@ one to open. | Run-version surfaces | `dw/runs.py`: `run_versions`, `dw/server/routes/gallery.py`, `dw/server/outputs.py`, `dw_mcp/tools_catalog.py`: `list_gallery`, `ui/src/lib/pages/GalleryPage.svelte` | Only `dw/runs.py` computes the number, and every other module reads it back. | `tests/test_runs.py::TestRunVersions::test_an_output_reference_can_name_a_run_by_its_version` | | Result subfolders | `dw/subfolders.py`: `subfolder_errors`, `dw/security.py`: `SUBFOLDER_PATTERN` | A step's `result.subfolder` is checked for shape at validation. | `tests/test_subfolders.py::TestValidationErrorsIntegration::test_validation_errors_reports_a_bad_subfolder` | | Template subfolder roles | `workflows/templates/`, `dw/workflows/` | Every saving step of a template is marked `final` or `intermediate`, with at least one `final`, and packaged builtins stay unmarked. | `tests/test_template_subfolders.py::test_every_saving_step_of_a_template_names_its_role`, `tests/test_template_subfolders.py::test_the_packaged_builtins_stay_unmarked` | +| JSON records beside media | `dw/media_types.py`: `JsonRecord`, `dw/result.py`: `Result.save_artifact` | A dict marked `JsonRecord` is saved whole as one `.json`, whatever content type the step declares, and is never split into one file per key. | `tests/test_face_track.py::test_json_record_is_saved_whole_under_a_video_content_type`, `tests/test_face_track.py::test_the_task_result_saves_crops_as_video_and_track_as_one_json` | +| Reading a JSON record back | `dw/locations.py`: `load_json_record` | A task reads a saved JSON record (`fit_to_model`'s, `crop_face_track`'s) only through `load_json_record`: the path is confined like any media path, then the extension must be `.json`, then the size is checked against `MAX_JSON_SIZE`, and only then is it parsed. `fit._read_fit` and `face_track._read_track` call it, never `open`. | `tests/test_locations.py::TestLoadJsonRecord::test_traversal_out_of_the_roots_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_a_non_json_extension_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_an_oversized_file_is_refused_before_it_is_read`, `tests/test_fit_to_model.py::TestRestore::test_fit_path_with_traversal_is_refused` | | Shots in a joined video | `dw/shots.py` | Every `AudioVideo` constructor decides what to do with `shots`, and the sample spans are measured from the joined waveform. | `tests/test_shots.py::test_every_audio_video_constructor_site_is_decided`, `tests/test_shots.py::TestConcatVideosShots::test_an_overrun_track_is_measured_not_derived` | -| Audio level of a deliverable | `dw/audio_qc.py`: `warn_without_headroom`, `warn_if_written_above_full_scale` | A muxed video keeps the pre-encode headroom prediction and falls back to it only when the post-write probe cannot measure the file. | `tests/test_result.py::TestNoHeadroom::test_a_clean_video_mux_drops_the_stale_prediction`, `tests/test_result.py::TestNoHeadroom::test_an_unprobeable_video_mux_falls_back_to_the_prediction`, `tests/test_result.py::TestTheWrittenLevel::test_a_file_that_decodes_above_full_scale_warns` | -| Template audio-level placement | `workflows/templates/minimax/music-video.json`, `workflows/templates/minimax/music.json` | `normalize_audio` (-3 dBFS) acts only on the track that goes into the final file, so the slices that condition the shots are untouched. | — | +| Overlap windows of a long video | `dw/tasks/windows.py`: `window_video`, `window_span` | Window `i` covers source frames `[i*stride - overlap, i*stride + stride)`, its synthetic frames repeat the first or last frame, and its audio is cut on the source's own `frames_to_samples` boundaries, so adjacent windows' strides tile the track with no sample lost or repeated. | `tests/test_window_video.py::TestAudio::test_strides_tile_the_source_track_exactly`, `tests/test_window_video.py::TestRealPath::test_a_path_argument_keeps_its_audio`, `tests/test_window_video.py::TestFileMatchesInMemory::test_every_window_equals_the_in_memory_window`, `tests/test_read_frame_range.py` | +| Keep a span of a video's frames and audio | `dw/tasks/trim.py`: `trim_video`; `dw/shots.py`: `trimmed_shots` | Frames `[start_frame, start_frame + num_frames)` are kept and the audio is cut on `frames_to_samples` of each end at the track's own rate, so consecutive trims tile the track. A span past the clip's end is refused, never shortened. Shots are clipped to the span, sample side cleared. | `tests/test_trim_video.py::TestFramesAndAudio::test_consecutive_trims_tile_the_track`, `tests/test_trim_video.py::TestShots::test_shots_are_clipped_across_the_boundary` | +| Join processed windows back to the source's length | `dw/tasks/windows.py`: `join_windows`; `dw/task_domains.py`: `window_count`, `window_count_problem`, `join_windows_errors` | Output is exactly the source's frames, each once: later windows' first `overlap` frames blend over the previous window's last on the open ramp `dissolve_videos` uses. The window count is `window_count`'s `ceil(source_frames / stride)` and nothing else (it is also `window_video`'s last index plus one). Audio is the source's own track; shots are one per window with `overlap_frames` on each seam, their sample side laid over that track by `dw/shots.py`'s `remeasured_shots` (the same rounding as `frames_to_samples`, so the spans are cumulative). | `tests/test_join_windows.py::TestRoundTrip::test_unprocessed_windows_rejoin_to_the_source`, `tests/test_join_windows.py::TestShots::test_spans_partition_the_frames_and_tile_the_audio`, `tests/test_join_windows.py::TestUint8Output::test_matches_the_float32_accumulator_exactly` | +| A `join_windows` list of the wrong length, before the run | `dw/window_count_errors.py`: `window_count_errors`, wired into `validation_errors` | Owned by the task, never a template: any `join_windows` step whose `source` is probeable header-only (`asset:`/`output:` or a readable literal path), with literal `num_frames`/`overlap` and an expanded `videos` list, is checked against `window_count_problem` - the same rule the run refuses with. Anything not knowable stays silent for the run. | `tests/test_window_count_errors.py` | +| Fit to a model and restore | `dw/tasks/fit.py`: `fit_to_model`, `restore_to_source` | fit_to_model's record is the whole of what restore_to_source needs: the scale is inferred from the output's size and must be uniform, and restore returns the source's size times that scale and at most its frame count. | `tests/test_fit_to_model.py` | +| Shared image ops | `dw/tasks/image_ops.py`: `LUMA_WEIGHTS`, `luma`, `split_alpha`, `join_alpha`, `per_frame` | `grade`, `finish` and `lut` take luma (float64 Rec. 709 weights, cast to the input's dtype) and alpha handling from `image_ops`, never from each other, so an RGBA input comes back RGBA with its alpha untouched; `lut.LUMA_WEIGHTS` is a re-export. An image command runs over a video or frame list only through `per_frame`. A media file is read by `task._load_media` with `fetch_image(keep_alpha=True)`, so a file with transparency arrives RGBA; a pipeline input still flattens to RGB. | `tests/test_image_ops.py::TestLuma::test_lut_re_exports_the_one_float64_weights`, `tests/test_image_ops.py::TestAlpha::test_rgba_round_trip_keeps_the_alpha_bytes`, `tests/test_image_ops.py::TestOutputsAreByteIdentical::test_every_command_matches_the_pre_refactor_bytes`, `tests/test_image_ops.py::TestPerFrame::test_a_video_comes_back_as_a_video_with_the_same_frames_and_audio`, `tests/test_finish.py::TestRgbaFileKeepsAlpha::test_a_file_path_comes_back_rgba_with_its_alpha` | +| Audio level of a deliverable | `dw/audio_qc.py`: `warn_without_headroom`, `warn_if_written_above_full_scale` | A muxed video keeps the pre-encode headroom prediction and falls back to it only when the post-write probe cannot measure the file. A result a later step resets the level of (`consumed_by_normalizer`, `dw/step_cache.py`) draws neither the post-write warning nor the fallback. | `tests/test_result.py::TestNoHeadroom::test_a_clean_video_mux_drops_the_stale_prediction`, `tests/test_result.py::TestNoHeadroom::test_an_unprobeable_video_mux_falls_back_to_the_prediction`, `tests/test_result.py::TestNoHeadroom::test_a_video_a_join_level_matches_draws_no_headroom_warning`, `tests/test_result.py::TestTheWrittenLevel::test_a_file_that_decodes_above_full_scale_warns` | +| Template audio-level placement | `workflows/templates/minimax/music-video.json`, `workflows/templates/minimax/music.json` | `normalize_audio` (-3 dBFS) acts only on the track that goes into the final file, so the slices that condition the shots are untouched. `music-video` slices each shot `lead_frames` early and `trim_video` drops the run-up before the join. | — | ## Validation, plan and cost | Concept | Owner | Rule | Enforced by | | --- | --- | --- | --- | | Validation registry | `dw/validation.py`: `ERROR_CHECKS`, `WARNING_CHECKS`, `run_checks` | A check is one `Check` in a registry, and a check that raises becomes one `internal` finding while the rest still run. | `tests/test_validation.py::TestExceptionPolicy::test_a_raising_check_is_one_internal_error_and_the_rest_still_run` | -| Step-value checks | `dw/step_value_checks.py`: `fps_errors`, `null_media_errors`, `select_errors` | The only caller of these checks is the error registry. | `dw/validation.py`: `ERROR_CHECKS` | +| Step-value checks | `dw/step_value_checks.py`: `fps_errors`, `null_media_errors`, `select_errors`, `chain_prompts_errors` | The only caller of these checks is the error registry. | `dw/validation.py`: `ERROR_CHECKS` | | Admission | `dw/server/admission.py`: `admit` | Validate, submit, rerun and enhance load and check a request once, and `JobManager.submit` does not check it again. | `tests/test_admission.py::test_a_submit_expands_once` | -| Variable constraints | `dw/variable_constraints.py`: `apply_constraints` | A model's rule about a value is declared in the workflow's `variable_constraints`, and it is checked at validation and again before anything loads. | `tests/test_variable_constraints.py::TestTheGrid::test_without_snap_an_off_grid_value_is_refused_not_rounded`, `tests/test_variable_constraints.py::TestTheRunTimePass::test_an_illegal_value_is_refused_before_anything_loads` | +| Variable constraints | `dw/variable_constraints.py`: `apply_constraints`; the grid arithmetic `aligned` / `aligned_down` | A model's rule about a value is declared in the workflow's `variable_constraints`, and it is checked at validation and again before anything loads. `plan_cuts` (and its static check) rounds render lengths through `aligned` / `aligned_down`, never its own copy, so a planned `num_frames` is one the template's constraint accepts; `crop_face_track` pads its crops to the step's `modulus`/`remainder` grid through `aligned` too (`padding_to_grid`). | `tests/test_variable_constraints.py::TestTheGrid::test_without_snap_an_off_grid_value_is_refused_not_rounded`, `tests/test_variable_constraints.py::TestTheGrid::test_plan_cuts_rounds_through_the_constraint_owner`, `tests/test_face_track.py::TestPadding::test_padding_to_grid_rounds_through_the_constraint_owner`, `tests/test_variable_constraints.py::TestTheRunTimePass::test_an_illegal_value_is_refused_before_anything_loads` | | Bound acknowledgement | `dw/plan.py`: `fingerprint`, `dw/server/admission.py`: `check_bound_acknowledgement` | A bound `acknowledged_cost` is refused with 409 when the plan's fingerprint or its required downloads changed. | `tests/test_server.py::TestBoundRerun::test_a_rerun_bound_to_a_stale_plan_is_refused`, `tests/test_server.py::TestBoundAcknowledgement::test_a_matching_fingerprint_queues` | | Per-repo gating | `dw/plan.py`: `build_plan` | Each gated repo, including a `loras` entry's repo, is probed and reported on its own. | `tests/test_plan.py::TestDownloadsRequired::test_a_gated_repo_this_token_lacks_access_to_is_blocked` | -| Observed cost | `dw/server/observed_cost.py`: `declared_drivers` | `observed` comes only from this box's finished jobs and never writes `cost`. Cold and warm runs are reported separately. | `tests/test_observed_cost.py::TestColdIsNotWarm::test_the_two_are_reported_separately_each_with_its_runs` | +| Observed cost | `dw/server/observed_cost.py`: `declared_drivers`, `ObservedCosts._on_this_card`, `ObservedCosts.card` | `observed` comes only from this box's finished jobs on the named card (the one `route` names, for `plan.estimate`), else the server's own card; `ObservedCosts.card` owns a card's (kind, name, default). Rows are matched by card name whatever index they ran on (a job from before jobs carried a `device` counts only while the server is on the default card), and never writes `cost`. Cold and warm runs are reported separately. | `tests/test_observed_cost.py::TestColdIsNotWarm::test_the_two_are_reported_separately_each_with_its_runs`, `tests/test_observed_cost.py::TestByCard::test_only_rows_from_this_card_count_whichever_index_they_had`, `tests/test_observed_cost.py::TestByCard::test_rows_from_before_jobs_carried_a_device_do_not_count_off_the_default_card` | | The estimate | `dw/plan.py`: `estimate`, `_tempered` | `plan.estimate` quotes the observed figure ahead of the curated one, and blends it toward the curated figure below three runs. | `tests/test_plan.py::TestObservedEstimate::test_history_beats_a_curated_figure`, `tests/test_plan.py::TestLowConfidenceObservedEstimate::test_a_single_run_blends_toward_the_curated_figure` | | Cost drivers | `dw/server/observed_cost.py`: `declared_drivers` | A cost driver that names no declared variable is dropped. | `tests/test_observed_cost.py::TestComparability::test_a_driver_naming_no_variable_is_dropped`, `tests/test_observed_cost.py::TestTheCatalogsDriversAreReal::test_every_declared_driver_is_a_variable_of_its_workflow` | | Raw workflow GET | `dw/server/routes/library.py`: `get_workflow` | The raw workflow GET is served verbatim, with no `observed`, because the editor saves what it reads. | — | -| VRAM projection | `dw/vram_estimate.py` | A template's `vram_estimate` is projected per step after `for_each` expansion, and only the largest step over the ceiling is reported. | `tests/test_vram_estimate.py::test_only_the_largest_over_ceiling_step_is_reported_once_per_cost_entry` | +| VRAM projection | `dw/vram_estimate.py`, `dw/guides.py`: `guide_lengths` | A template's `vram_estimate` is projected per step after `for_each` expansion, and only the largest step over the ceiling is reported. With `bytes_per_guide_voxel` declared, each guide the step lays in (its own non-null `guides` plus a guide chain's) adds its snapped frames times the canvas; an unprobeable clip is charged at `num_frames`, and validate, admission and run project the same number. | `tests/test_vram_estimate.py::test_only_the_largest_over_ceiling_step_is_reported_once_per_cost_entry`, `tests/test_vram_guides.py::test_validate_apply_and_required_agree_on_a_guided_definition` | | H3 Ref2VA VRAM numbers | `workflows/templates/minimax/` | Every Ref2VA template declares `base_gb` 16.0, `bytes_per_voxel` 28.71 and `gb_per_reference` 1.0. | `tests/test_h3_vram_ceiling.py::test_every_ref2va_minimax_template_declares_gb_per_reference` | | Inherited VRAM ceiling | `dw/vram_inheritance.py`, `dw/server/deps.py`: `ceiling_index` | A workflow with no `vram_estimate` is matched against the catalog by pipeline identity, and is warned, never refused. | `tests/test_vram_inheritance.py::test_every_template_declaring_one_identity_declares_the_same_numbers` | | H3 adapter partition | `dw/adapter_compatibility.py` | An FL2VA LoRA on a reference step is refused, and a LoRA whose file name says neither `ref2v` nor `fl2v` is warned. | `tests/test_h3_adapters.py::TestWhatIsRefused::test_a_keyframe_adapter_on_the_reference_path`, `tests/test_h3_adapters.py::TestWhatIsRefused::test_an_unrecognised_name_is_a_warning_not_a_refusal` | +| H3 audio hold, refine and guide layout | Three modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules asked before validation and before the call; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants), `dw/pipeline_processors/h3_hold.py` the hold and refine blocks (`insert_audio_hold`, `core_denoise_sequences`, `refines`, `encode_audio_span`), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `splice_guide_rows`, `insert_guides`), which imports `h3_hold` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline.py` adds its blocks for a `guides` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/hold_audio.py`: `hold_audio_errors`, `refine_strength_errors` | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and they are no-ops without `hold_audio` / `refine_strength`. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers` | | IC-LoRA reference scale | `workflows/templates/ltx2/` | The three conditioning templates run at `reference_downscale_factor: 1`, and the two upscalers at 2. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_reference_is_encoded_at_the_output_resolution`, `tests/test_ltx2_ic_loras.py::TestTheGenerativeUpscaleMatchesTheSameCard::test_it_loads_the_upscaler_at_factor_two` | | IC-LoRA numbers | `workflows/templates/ltx2/` | Every number in the IC-LoRA templates comes from the vendor's model card. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_strength_is_the_cards_default`, `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_defaults_are_the_trained_bucket` | | IC-LoRA prompt genre | `prompts/ltx2/` | A stored prompt tagged `ic-lora` is checked against its trained caption form, not against the T2V paragraph rule. | `tests/test_ltx_prompt_library.py::test_an_ic_lora_prompt_is_in_its_trained_form` | @@ -99,7 +109,8 @@ one to open. | No seed, no cache | `dw/workflow_run.py`: `prepare_run` | A workflow that sets no `seed` skips the cache. | `tests/test_workflow_step_cache.py::TestCacheHits::test_an_unseeded_workflow_has_no_hits` | | A different result | `dw/server/routes/jobs.py`, `dw_mcp/tools_jobs.py`: `rerun_job` | `rerun` with `new_seed` draws a fresh seed into the workflow's seed variable. | `tests/test_rerun_new_seed.py::test_a_rerun_with_a_new_seed_draws_one_into_that_variable` | | Worker protocol | `dw/worker_protocol.py`: `parse_reply` | Every command and reply is a frozen dataclass that travels as a wire dict. | `tests/test_worker_messages.py::test_from_wire_inverts_to_wire`, `tests/test_worker_messages.py::test_an_unknown_reply_type_is_kept_whole_rather_than_raised` | -| Persistent worker | `dw/worker.py`, `dw/worker_manager.py`, `dw/serve.py` | Jobs run in one spawned worker that keeps models loaded, so a change to engine code needs a server restart. | — | +| Persistent worker | `dw/worker.py`, `dw/worker_manager.py`, `dw/serve.py` | Jobs run in spawned workers (one per card) that keep models loaded, so a change to engine code needs a server restart. | — | +| Worker card pinning | `dw/devices.py`: `worker_environment`, `pinned_environment`, `resolve_serve_devices`, `check_device_present`, `label_of` | A worker named with a CUDA index is spawned with `CUDA_VISIBLE_DEVICES` set to that card and `DW_DEVICE=cuda` (a bare `cuda` pins nothing), so index-0 reads inside it measure its own card; `--devices`/`devices` takes one entry per worker (a card named twice, or a bare `cuda` in a list of several, is refused), a card the box lacks is refused at startup naming the cards present, and a job stores the card it ran on as `device_ordinal` and `device_card` (#693), job and history alike: rerun affinity reads the ordinal, observed cost buckets by the card, and the `device` label `"cuda:N "` is only `label_of` joining the two - nothing splits a label back apart. | `tests/test_worker_manager.py::TestWorkerPinning::test_a_cuda_index_pins_the_worker_through_the_environment`, `tests/test_worker_manager.py::TestWorkerPinning::test_a_bare_cuda_sets_no_pin`, `tests/test_serve_main.py::TestConfigureDevices::test_a_refusal_exits_with_code_2`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_the_machine_lacks_is_refused_naming_the_cards_present`, `tests/test_serve_main.py::TestResolveServeDevices::test_two_entries_are_both_kept`, `tests/test_serve_main.py::TestResolveServeDevices::test_a_card_named_twice_is_refused` | | Failed-run reporting | `dw/worker.py`, `dw/worker_protocol.py`: `Failed`, `Cancelled` | A failed or cancelled run's reply still carries the manifest of the steps that ran. | `tests/test_worker_execute.py::test_failure_carries_the_manifest_of_the_steps_that_ran`, `tests/test_worker_execute.py::test_cancellation_carries_the_manifest_too` | | Failure path redaction | `dw/path_redaction.py`: `redact_paths`, called by `dw/worker.py` | A failed run's message and traceback name a file under the job's asset roots or output directory by its `asset:`/`output:` reference, never its absolute server path. | `tests/test_worker_execute.py::test_a_failure_names_an_asset_by_reference_not_by_server_path`, `tests/test_path_redaction.py` | | Run-time warnings | `dw/events.py`: `emit_warning` | A warning found at run time is emitted as an event, so it reaches the caller and not just the log. | `tests/test_concat_videos.py::TestWarningsReachTheCaller::test_the_level_spread_warning_is_emitted_as_an_event` | @@ -111,6 +122,7 @@ one to open. | Opening media | `dw/media.py` | This is the only module that calls `av.open`. It imports no torch and no step types. | `tests/test_media_layering.py::test_av_open_appears_only_in_media`, `tests/test_media_layering.py::test_media_imports_no_step_types` (no torch: —) | | Signal processing | `dw/dsp.py` | It measures and transforms waveforms, decides nothing, and imports nothing from `dw`. | `tests/test_media_layering.py::test_dsp_imports_nothing_from_dw` | | Assessment probes | `dw/tasks/assess.py`, `dw/assessment_rules.py` | Exactly three registered commands are assessment probes. Each measures a finished file, and every rule names a real probe field. | `tests/test_assessment_rules.py::TestRuleProbesAreRealCommands::test_assessment_is_exactly_the_three_probes`, `tests/test_assessment_rules.py::TestRuleProbesAreRealCommands::test_every_rule_names_a_registered_json_command` | +| Script check | `dw/tasks/script_check.py`: `check`; `dw/script_lines.py`: `parse_lines`, `parse_shots`, `shot_names_error` | `check_script` aligns a take's heard words to its expected lines and guards out words heard over silence and repetition loops (Whisper's hallucination over music). It builds its findings with `assessment_rules.finding()` and reads the guard's floor and window (`DEAD_AIR_FLOOR_DBFS`, `DEAD_AIR_WINDOW`) rather than copying them. It is not an assessment probe and adds nothing to `RULES`. Its literal `lines` is refused at validation through `task_domains` (`script_lines_errors`), which calls the same `parse_lines` the task runs, and a literal `shots` through the same `parse_shots` and `shot_names_error`; the parsers live in `dw/script_lines.py`, not the task, so that `task_domains` reaches them without importing `script_check` (the `import_cycles` ratchet in `scripts/arch_metrics.py`). Its shot map comes from `assess.resolve_shots`, the one the probes use, never a copy; it places each shot on the soundtrack with `assess.sample_span`, decides repeated shot names with `shots.duplicate_shot_names`, and asks `locations.is_http_url` whether the take is a URL. | `tests/test_script_check.py::TestRegistration::test_registered_as_json_command_without_assessment_flag`, `tests/test_script_check.py::TestEmptyLines::test_words_only_over_silence_are_discarded_not_found`, `tests/test_script_check.py::TestLinesErrors::test_literal_with_a_colon_is_not_a_reference` | ## Security and architecture guardrails @@ -118,7 +130,8 @@ one to open. | --- | --- | --- | --- | | Path validators | `dw/security.py`: `validate_path`, `validate_workflow_path`, `validate_output_path` | Filesystem access goes through a path validator that is given a base directory. | `tests/test_security.py::test_path_validation`, `.github/codeql/dw-security/DwPathInjection.ql` | | URL and argument validators | `dw/security.py`: `validate_url`, `sanitize_command_args` | A URL must be http or https, and a subprocess argument must not carry a shell metacharacter. | `tests/test_security.py::TestValidateUrl::test_dangerous_schemes_are_rejected`, `tests/test_security.py::TestSanitizeCommandArgs::test_every_shell_metacharacter_is_rejected` | -| Locations from a workflow | `dw/locations.py` | A local path in a workflow's media arguments is confined to a known root, and a URL must not name a host inside the deployment. | `tests/test_locations.py::TestMediaPathContainment::test_absolute_path_outside_every_root_is_refused`, `tests/test_security_ssrf.py::TestInternalAddressSpellings::test_a_name_that_resolves_inside` | +| Locations from a workflow | `dw/locations.py` | A local path in a workflow's media arguments (named like their media, or declared by the task's `@register_command(media_arguments=...)`, read through `TASK_MEDIA_ARGUMENTS`) is confined to a known root, a location written as a URL is one `security.validate_url` accepts, and a URL must not name a host inside the deployment. A host that does not resolve is refused. The requests themselves are `dw/outbound.py`'s. | `tests/test_locations.py::TestMediaPathContainment::test_absolute_path_outside_every_root_is_refused`, `tests/test_locations.py::TestMediaPathContainment::test_a_url_that_is_not_http_is_refused`, `tests/test_locations.py::TestMediaPathContainment::test_an_upper_case_https_url_is_not_refused_as_another_scheme`, `tests/test_security_ssrf.py::TestInternalAddressSpellings::test_a_name_that_resolves_inside`, `tests/test_locations.py::TestTaskMediaArguments::test_an_unreadable_source_is_an_error_at_its_path` | +| Outbound requests | `dw/outbound.py`: `safe_get`, `safe_post` | Every outbound request is validated per hop against `dw/locations.py`'s policy, resolved once and dialed at the address the policy checked, capped and timed. No other module under `dw/` makes an HTTP request or builds a session or adapter. | `tests/test_outbound.py::TestSafeRequest::test_one_lookup_per_hop_and_the_dial_is_the_checked_address` | | Trust gate | `dw/trust.py`: `require_trusted_dotted_name` | In an untrusted workflow (the default), a dotted type outside the allowlist is a validation error before anything is imported. | `tests/test_security_trust_gate.py::TestValidationRefusesBeforeImport::test_a_dotted_type_is_a_validation_error` | | CodeQL path-injection model | `.github/codeql/dw-security/DwPathSanitizers.qll`, `.github/workflows/codeql.yml` | The local pack models the `dw/security.py` validators as sanitizers, so a validator that moves is re-modelled in the same commit. | `.github/codeql/dw-security/DwPathInjection.ql` | | Archives never follow links | `dw/server/outputs.py`: `zip_download` | A gallery listing drops a symlink that leaves its root, and an archive skips one. | `tests/test_security_symlinks.py::TestOutputs::test_the_archive_route_does_not_follow_the_link`, `tests/test_security_symlinks.py::TestOutputs::test_the_gallery_listing_does_not_enumerate_the_link` | @@ -131,10 +144,14 @@ one to open. | --- | --- | --- | --- | | App factory | `dw/server/app.py`: `create_app` | `create_app` builds the JobManager and `app.state`, installs the middleware and registers the routes, and all state lives in the JobManager. | — | | Routers | `dw/server/routes/*.py`, `dw/server/routes/__init__.py`: `ROUTERS`, `include_routers`, `include_file_routes` | Routers register in `ROUTERS` order, because a greedy `{name:path}` route must come after its specific siblings. The files router comes later, after `/mcp`. | `tests/test_server_downloads.py::test_download_workflow_sets_content_disposition_attachment` | -| Job queue | `dw/server/jobs.py`: `JobManager` | One runner thread runs jobs FIFO on the single worker. A job carries its own `output_dir`, `asset_dir` and `workflow_dir`, so it stays in its workspace. | `tests/test_server_workspaces.py::TestRunning::test_rerun_stays_in_the_workspace_it_ran_in` (FIFO: —) | +| Job queue | `dw/server/jobs.py`: `JobManager`; `dw/server/job_results.py`: `consume_results` | A dispatcher thread hands queued jobs, in queue order, to the pool's workers (`Worker pool dispatch`); each worker runs one job at a time, and `consume_results` folds that worker's replies into the job, naming files relative to the job's own `output_dir`. A job carries its own `output_dir`, `asset_dir` and `workflow_dir`, so it stays in its workspace. | `tests/test_server_workspaces.py::TestRunning::test_rerun_stays_in_the_workspace_it_ran_in` (FIFO: —) | +| Worker pool dispatch | `dw/server/jobs.py`: `JobManager._next_dispatch`; `dw/server/pool.py`: `WorkerSlot`, `check_fits`, `largest_ceiling_gb`; `dw/vram_estimate.py`: `required_vram_gb`; `dw/server/admission.py`: `pool_capacity` | `--devices` runs one worker per card; a queued job goes to a free card that fits what it needs, a job that fits no free card keeps its place while a smaller one behind it takes the card, a job bigger than every card is refused at submit, and cancel or a crash touches only the worker running that job. On a backend no `cost` entry describes, a declared `vram_estimate` is held at admission to the pool's largest card, so a job the first card cannot hold is not refused when a larger card can hold it. | `tests/test_worker_pool.py::TestConcurrency::test_two_workers_run_two_jobs_at_once`, `tests/test_worker_pool.py::TestBackfill::test_a_job_too_big_for_the_free_card_waits_while_a_smaller_one_takes_it`, `tests/test_worker_pool.py::TestFitAtSubmit::test_a_job_too_big_for_every_card_is_refused_naming_the_largest`, `tests/test_worker_pool.py::TestCancel::test_cancel_reaches_the_worker_running_the_job`, `tests/test_worker_pool.py::TestCrashIsolation::test_a_crashed_worker_fails_only_its_own_job`, `tests/test_worker_pool.py::TestOomRanking::test_the_later_started_worker_is_ranked_first`, `tests/test_worker_pool.py::TestFitAtSubmit::test_a_job_only_the_larger_card_holds_is_admitted` | +| Card affinity | `dw/server/pool.py`: `choose_slot`; `dw/server/jobs.py`: `JobManager.route`; `dw/worker_protocol.py`: `workflow_identity` | Among the free cards a job fits it goes to the rerun's original card, else the card whose worker last ran its workflow identity (`workflow_identity`, the one rule the worker caches by too), else one with no worker running, else the first in `--devices` order; a cache probe and `plan.estimate` (`priced_for`) ask about the card `route` names, and memory is read and cleared per card. | `tests/test_worker_pool.py::TestIdentityAffinity::test_a_second_run_lands_on_the_card_that_ran_the_workflow_first`, `tests/test_worker_pool.py::TestRerunAffinity::test_a_rerun_lands_on_the_original_card_when_it_is_free`, `tests/test_worker_pool.py::TestProbeRouting::test_the_probe_goes_to_the_worker_the_job_would_be_routed_to`, `tests/test_worker_pool.py::TestMemoryPerCard::test_a_named_card_running_a_job_is_refused_though_the_other_is_idle` | +| Per-card memory and cache | `dw/server/worker_memory.py`: `memory_status`, `clear_memory`, `probe_cache` | Each card keeps its own last memory reading; an idle card's worker is asked live, a busy one answers with the run's last report and says why it is not live, and a clear leaves a card running a job alone. Every request takes the card's lock with a bounded wait, and a card's lock is never taken while the JobManager's lock is held. | `tests/test_worker_pool.py::TestMemoryPerCard::test_a_named_card_running_a_job_is_refused_though_the_other_is_idle`, `tests/test_worker_pool.py::TestProbeRouting::test_the_probe_goes_to_the_worker_the_job_would_be_routed_to`, `tests/test_server.py::test_memory_says_why_a_reading_is_not_the_worker_s` | | Job-to-run link | `dw/server/job_record.py`, `dw/server/jobs.py`: `JobManager.realized` | A job records `run_id`, `run_dir` and `run_version`, and `JobManager.realized` reads that run's `workflow.json`, confined to the output root. | `tests/test_server_jobs.py::test_realized_reads_the_file_the_run_wrote`, `tests/test_server_jobs.py::test_realized_refuses_a_run_dir_that_escapes_the_output_root` | | Job history | `dw/server/job_history.py`: `JobHistory` | Finished jobs persist in `jobs.sqlite`, in WAL mode, so the Jobs view survives a restart. | `tests/test_server.py::test_job_history_survives_restart_and_reruns`, `tests/test_job_history_wal.py::test_the_jobs_database_uses_wal_mode` | | HTTP security | `dw/server/http_security.py`: `install_middleware`, `query_token_ok` | Every API route needs the token, and only a route marked `query_token_ok` takes it as `?token=`. | `tests/test_security_auth.py::TestTheTokenGate::test_every_api_spelling_needs_the_token`, `tests/test_security_auth.py::TestTheTokenGate::test_the_query_token_is_refused_where_it_is_not_allowed` | +| Prompt enhancement | `dw/server/enhancers.py`: `PRESETS`, `_T2I_SYSTEM_PROMPT` | `PRESETS` is the Enhance panel's menu (label, curated LLMs, preselect hints); a preset's family-specific knowledge, its spec, lives in a builtin under `dw/workflows/`, never in Python. `t2i`'s generic, family-agnostic system prompt stays in Python. A third preset moves the menu into builtin metadata (#696). | `tests/test_prompt_library.py::test_every_enhancer_preset_targets_a_known_family`, `tests/test_server.py::TestEnhance::test_presets_are_listed`, `tests/test_server.py::TestEnhance::test_enhance_workflows_are_schema_valid`, `tests/test_server.py::TestEnhance::test_an_unknown_preset_is_a_client_error` | ## MCP diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index 6ee609e0..0920fd0d 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -47,7 +47,7 @@ pyproject; install.ps1 installs it explicitly on Windows. ## Platform-Specific Dependencies -**All platforms (core ML):** peft, transformers, accelerate, safetensors, controlnet_aux, sentencepiece, torchsde, torchao, optimum-quanto, gguf, kornia, ftfy, sdnq, spandrel, facexlib (spandrel + facexlib back the `upscale` and `restore_faces` tasks) +**All platforms (core ML):** peft, transformers, accelerate, safetensors, controlnet_aux, sentencepiece, torchsde, torchao, optimum-quanto, gguf, kornia, ftfy, sdnq, spandrel, facexlib (spandrel + facexlib back the `upscale` and `restore_faces` tasks). On first use, `restore_faces` has facexlib download its face detection and parsing weights (about 110 MB) from facexlib's own GitHub releases into the facexlib package directory. It is the one model download that bypasses the Hugging Face cache, dw's download manager and the `plan.downloads_required` estimate. **All platforms (utilities):** fastapi, uvicorn (the `dw.serve` HTTP server and web UI), av, aiohttp, matplotlib, opencv-python-headless, concurrent-log-handler, qrcode, protobuf, imageio, imageio-ffmpeg, beautifulsoup4, soundfile, jsonschema, black, python-dotenv diff --git a/docs/MCP.md b/docs/MCP.md index 0437a64c..f1aa1945 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -200,6 +200,11 @@ fetching every candidate's definition. Reusing a stored workflow is a preference, not a rule; `run_workflow` still takes an `inline_workflow` for a request nothing on disk covers. +The instructions also say the engine runs **one job per GPU** - one worker +per `dw.serve --devices` card, each running its own job from the one shared +queue - and point at `get_health`'s `workers` for what each card is running. +A job record's `device` (`get_job`, `list_jobs`) names the card it ran on. + A request usually names a *subject* ("a lego movie trailer set in the marvel universe") while the catalog is written in *shapes* - a single image, an image set, one shot, a multi-shot cut sequence, video with speech. Nothing in a @@ -216,14 +221,14 @@ when no single workflow covers it. | `get_workflow(name, variables_only=False)` | `name` | Get one stored workflow's full JSON definition. `variables_only=true` answers with just its variables and their defaults (long strings cut to 200 characters, the cut ones named in `truncated`, including strings inside a list default, named like `shots[0].prompt`) — the cheap way to confirm what a variable defaults to | | `get_schema(section=None)` | `section` | Get the JSON schema every workflow definition must satisfy. Whole it is ~8.6k tokens, so name the part you need: `section` takes `steps`, `pipelines`, `tasks`, `result`, `variables` or `configuration` and answers `{section, sections, elsewhere, schema}` - `elsewhere` says which section holds each definition the fragment still `$ref`s. An unknown section is a 404 naming the ones there are | | `list_pipelines()` | — | List every diffusers pipeline class this installation provides | -| `get_pipeline_signature(name)` | `name` | Get a pipeline's real call arguments | +| `get_pipeline_signature(name)` | `name` | Get a pipeline's real call arguments; for a modular pipeline class (e.g. `MiniMaxH3ModularPipeline`), its block graph's inputs | | `list_classes(kind)` | `kind` | List class names of one kind: pipelines, models, schedulers, or quantization | | `get_class(name, target="init")` | `name`, `target` (`init`\|`call`\|`load`) | Get a class's argument schema from the entry point a workflow reaches it by: `init` the constructor (quantization configs, schedulers), `call` a pipeline's `__call__`, `load` `from_pretrained` plus the curated loading knobs | | `list_tasks()` | — | List every task command a workflow's task step can name | | `get_task(command)` | `command` | Get a task command's argument schema | | `list_models()` | — | List what the Hugging Face model cache holds, largest first | -| `get_memory()` | — | Get the worker's VRAM and RAM statistics. `gpu_*` is the card; `host_memory_rss_mb` / `host_memory_peak_rss_mb` are the worker process's resident and high-water host memory, beside the machine's `host_memory_total_mb` / `host_memory_available_mb` - read both, since an offloading workflow keeps its weights in host RAM and the card says little about what it holds (a host field is absent, not null, where the platform cannot measure it). `host_pinned_reserved_mb` / `host_pinned_allocated_mb`, when present, are torch's pinned-host cache - the staging buffers group offloading moves weights through, part of `host_memory_rss_mb` and invisible in every `gpu_*` figure, which is what a worker holding GB after releasing every model is usually holding (#98). `live: true` means `info` was measured now and is the worker's own memory; only live readings are comparable with each other. `live: false` with `info: null` (and `stale: false`) means nothing has been measured because nothing is resident; `live: false` with a populated `info` is a cached earlier reading, with `reason` (`job_running`, `worker_stopped`, `worker_busy`, `worker_unreachable`) and `age_seconds` - one cached while a job loads a model understates what is resident, so ask again when the server is idle rather than comparing it with a live figure | -| `get_health()` | — | Check that the server is alive, and which machine answered: `version`, `device`, whether a model process is currently resident (`worker_alive`), the job running now and the queue depth. `worker_alive: false` is the normal idle state on a server that has not run a job since startup or the last memory clear - not a fault - the on-demand worker starts with the next job (#206) | +| `get_memory()` | `device` | Get VRAM and RAM statistics: `workers` has one reading per card (the top level repeats the first card's), and `device` (`cuda:1`) asks for that card alone. `gpu_*` is the card; `host_memory_rss_mb` / `host_memory_peak_rss_mb` are the worker process's resident and high-water host memory, beside the machine's `host_memory_total_mb` / `host_memory_available_mb`; on Linux `host_memory_rss_anon_mb` / `host_memory_rss_file_mb` split the resident figure into memory the worker owns and mapped-file pages (memory-mapped checkpoint weights) the kernel can drop, so a loader that maps rather than copies raises the resident and peak figures without raising committed memory (#709) - read both, since an offloading workflow keeps its weights in host RAM and the card says little about what it holds (a host field is absent, not null, where the platform cannot measure it). `host_pinned_reserved_mb` / `host_pinned_allocated_mb`, when present, are torch's pinned-host cache - the staging buffers group offloading moves weights through, part of `host_memory_rss_mb` and invisible in every `gpu_*` figure, which is what a worker holding GB after releasing every model is usually holding (#98). `live: true` means `info` was measured now and is the worker's own memory; only live readings are comparable with each other. `live: false` with `info: null` (and `stale: false`) means nothing has been measured because nothing is resident; `live: false` with a populated `info` is a cached earlier reading, with `reason` (`job_running`, `worker_stopped`, `worker_busy`, `worker_unreachable`) and `age_seconds` - one cached while a job loads a model understates what is resident, so ask again when the card is idle rather than comparing it with a live figure | +| `get_health()` | — | Check that the server is alive, and which machine answered: `version`, `device`, whether any model process is resident (`worker_alive`), and the queue depth. The engine runs one job per GPU: `workers` has one entry per card - `device`, `name`, `vram_gb`, `alive`, its own `current_job`, and the worker's `host_memory_rss_mb` while it runs - and the top-level `current_job` is the longest-running job. `worker_alive: false` is the normal idle state on a server that has not run a job since startup or the last memory clear - not a fault - the on-demand worker starts with the next job (#206) | | `get_server_info()` | — | What this installation can do and where it keeps things: `device` (the accelerator a run will use), `version`, the `workspace` this session is working in and the workflow/asset/output/prompt `directories` of *that* workspace, the bind address and port, whether a token is required, and whether MCP is mounted. Check the device before authoring - a CUDA-only choice (bitsandbytes, `torch.compile`, flash attention) is not available on an `mps` or `cpu` server. `runtime` (#222) reports the Python version, torch version and the CUDA version torch was built against, the NVIDIA driver version (when `nvidia-smi` is reachable), and the installed versions of diffusers, transformers, accelerate, bitsandbytes, peft, safetensors and sentencepiece (`null` for one not installed) - for diagnosing an environment mismatch between boxes without shelling in | | `list_jobs(limit=20, status=None, workspace=None)` | optional `limit` (newest N), `status` (one state or a comma-separated set of `queued`, `running`, `succeeded`, `failed`, `cancelled`), `workspace` | List queued, running and recent jobs, **newest first**. Bounded by default: the unbounded listing was over a client's tool-result limit on a server with a few months of history, which made it a tool that could not be called at all. `total` says how many matched and `truncated`/`next` say so when the answer was cut - raise `limit` or narrow with `status`. Without `workspace`, a named workspace lists its own jobs and the default one lists every job the server holds | | `list_gallery(limit=50, subfolder=None, only_orphans=False, workspace=None, folder=None, version=None)` | `limit`, `subfolder`, `only_orphans`, `workspace`, `folder`, `version` | List generated output files, newest first. A name is `//`, where `` may sit in the subfolder the step chose (`final/episode.mp4`); each entry carries `folder` (the workflow) and `subfolder` (by convention `final` or `intermediate`, `''` when the step chose none, any path the workflow wrote otherwise), and `subfolder="final"` lists only deliverables. Each entry carries `run_id` and `version` - that run's ordinal among the workflow's runs, which is how one of several runs that wrote the same basename is named to a person: the web UI labels the same file `v5`. The number is assigned when the run opens and never renumbered, so deleting a run leaves a gap rather than sliding the rest down (a failed run, or a rerun that reused every step, leaves one too - it took a number and may have nothing to list), and it is `null` under the flat output layout, which has no runs. `folder` with `version` lists that one run's files, and `output:/v5/` names one in a workflow; every other tool takes `name`. Each entry also carries a ready-made `url`, already scoped to the workspace that made it - a hand-built `/outputs/` URL 404s for anything but the default workspace. `only_orphans=True` inverts the call: instead of files, it returns run directories with no media anywhere under them (`runs`, each `{name, mtime}`) - a run whose output was deleted before `delete_output` could remove it by name, or one that failed before writing anything; `subfolder` does not apply in this mode, and `name` is exactly what `delete_output` accepts (#170). `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | @@ -276,7 +281,7 @@ tools) rather than relying on the session pin. | `delete_workspace(name, acknowledged_cost=False)` | `name`, `acknowledged_cost` | Permanently delete a workspace and everything in it. Refuses without the acknowledgement, reporting what it would remove | | `list_assets()` | — | The input media on the server, each with the `asset:` reference a workflow argument carries. Look here before asking for a file - what a workflow needs may already be there. `libraries` names the roots searched and which are writable; `shadowed` lists names a nearer library hides | | `keep_output(name, asset_name=None, overwrite=False, shared=False, workspace=None)` | `name`, optional `asset_name`, `overwrite`, `shared`, `workspace` | Keep a generated file as an input asset under a stable `asset:` name, so a later workflow can rely on it. The copy happens on the server: nothing is downloaded or re-uploaded. `asset_name` may name a folder and takes the kept file's extension when it has none; `shared=true` keeps it in the library every workspace shares, which is where a recurring cast belongs. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | -| `upload_asset(file_path=None, content=None, asset_name=None, shared=False)` | exactly one of `file_path` or `content` (base64), optional `asset_name`, `shared` | Push an image, video or audio file into the server's asset library and get back its `asset:` reference. `file_path` is read from the machine the MCP server runs on, so this is how an input reaches a dw.serve running somewhere else - but only when the caller shares a filesystem with that machine. `content` is the alternative for an agent that does not: the bytes travel inline, base64-encoded, in the tool call itself, capped at 4MB (well under `file_path`'s 200MB) because inline bytes compete with the calling agent's own context budget rather than being a bulk-transfer path (#203). `asset_name` is required with `content` (there is no file name to infer one from) and, either way, stores it under a readable name (`cast/priya-voice.wav`) instead of a random one; `shared=true` puts it in the library every workspace shares | +| `upload_asset(file_path=None, content=None, asset_name=None, shared=False)` | exactly one of `file_path` or `content` (base64), optional `asset_name`, `shared` | Push an image, video or audio file, or a `.cube` 3D LUT for `apply_lut`, into the server's asset library and get back its `asset:` reference. `file_path` is read from the machine the MCP server runs on, so this is how an input reaches a dw.serve running somewhere else - but only when the caller shares a filesystem with that machine. `content` is the alternative for an agent that does not: the bytes travel inline, base64-encoded, in the tool call itself, capped at 4MB (well under `file_path`'s 200MB) because inline bytes compete with the calling agent's own context budget rather than being a bulk-transfer path (#203). `asset_name` is required with `content` (there is no file name to infer one from) and, either way, stores it under a readable name (`cast/priya-voice.wav`) instead of a random one; `shared=true` puts it in the library every workspace shares | | `delete_asset(name)` | `name` | Permanently remove one file from the asset library, by the name `list_assets` reports. Deletes from whichever library holds it - this workspace's own before the shared one; one from a read-only examples library is refused. Any workflow still carrying that `asset:` reference stops loading | | `save_workflow(name, workflow)` | `name`, `workflow` | Save a workflow into the server's writable workflow directory, overwriting any existing workflow of that name there. A name that currently resolves to a read-only source (an examples directory) is not overwritten - the copy lands in the writable directory and shadows it | | `delete_workflow(name)` | `name` | Permanently delete a stored workflow | @@ -312,7 +317,7 @@ references written in the same session. | `get_job_events(job_id, after=-1, limit=200)` | `job_id`, `after`, `limit` | Get a page of a job's progress events | | `wait_for_job(job_id, timeout_seconds=20)` | `job_id`, `timeout_seconds` | Block until a job reaches a terminal status, or `timeout_seconds` elapses. **One call blocks for at most the server's cap** — 55 seconds unless the deployment sets `DW_MCP_MAX_WAIT_SECONDS` higher, and the tool's description states the live value; a larger `timeout_seconds` is clamped, not honoured. Ask for the job's `plan.estimate` plus a margin: under the cap that is one call for the whole job, over it one call per cap. A deployment raises the cap only as far as its clients (and anything between them and the server) hold one silent HTTP request open - Claude Code's limit is `MCP_TOOL_TIMEOUT`. Every reply carries `waited_seconds`, `timeout_requested_seconds`, `timeout_applied_seconds` and `timeout_capped`, so a capped return is distinguishable from an elapsed one. Use instead of hand-polling `get_job`/`get_job_events` in a loop; if it returns `still_running: true`, call it again. Returns a slim job - status, warnings, error, `run_id` and `run_version` (the run's `v5`, as the gallery labels it), and the manifest once finished - without the arguments; `get_job` has those. A running job also carries `progress` (below) | | `cancel_job(job_id)` | `job_id` | Ask a queued or running job to stop | -| `clear_memory()` | — | Drop every loaded pipeline and the step cache, freeing VRAM/RAM immediately instead of waiting for the next job to evict one model for another. Also drops the step cache, so a seeded workflow that would otherwise reuse cached results regenerates on its next run. Refused with a 409 while a job is running or queued - the queue is FIFO, so wait for it to finish and retry rather than expecting this call to block until it does (#221) | +| `clear_memory()` | `device` | Drop every loaded pipeline and the step cache on each idle card - `device`'s alone when named - freeing VRAM/RAM immediately instead of waiting for the next job to evict one model for another. Also drops the step cache, so a seeded workflow that would otherwise reuse cached results regenerates on its next run. A card running a job is skipped (`workers` reports it `job_running`); refused with a 409 when no card asked about is idle, so wait and retry rather than expecting this call to block (#221, #677) | | `rerun_job(job_id, acknowledged_cost=False, new_seed=False)` | `job_id`, `acknowledged_cost`, `new_seed` | Queue a fresh job from a previous job's stored specification. Costs GPU time, so it passes the same gate as `run_workflow`. `new_seed=true` draws a fresh seed into the workflow's seed variable — without it a seeded workflow's rerun repeats its arguments exactly and the step cache serves the whole run from the earlier one's files (`reused: true`), generating nothing. `get_job_workflow`'s `seed_variable` says whether there is one - `acknowledged_cost` is `true` or the bound `{fingerprint, minutes, downloads}` from the validate plan; a 409 means the plan changed and the message carries the new estimate | | `move_job(job_id, direction)` | `job_id`, `direction` (`up`\|`down`\|`front`\|`back`) | Reorder a queued job | @@ -343,7 +348,7 @@ through is not a gate. | Tool | What it commits | | --- | --- | -| `run_workflow` | Minutes of GPU time; the engine runs one job at a time | +| `run_workflow` | Minutes of GPU time; the engine runs one job per GPU, so a run holds a card the queue could give another job | | `rerun_job` | The same run, from a stored spec | | `download_model` | Tens of gigabytes of network and disk | | `delete_model` | Cached weights, unrecoverably — getting them back means downloading again | diff --git a/docs/RECIPES_24GB.md b/docs/RECIPES_24GB.md index 70762424..6385aae5 100644 --- a/docs/RECIPES_24GB.md +++ b/docs/RECIPES_24GB.md @@ -185,7 +185,8 @@ same seed: fur, branches and snow texture resolve where the upsample-only frame soft blur. About eight warm minutes, three and a half of them writing the full-size clip. The same refine on a clip dw did not make is [refine-clip.json](../workflows/templates/ltx2/refine-clip.json): the upsampler encodes -the source itself, and the source's soundtrack is paired back on. +the source itself, fitted to the working size first and restored to exactly twice the +source's size and length after, and the source's soundtrack is paired back on. **Examples:** [text-to-video.json](../workflows/templates/ltx2/text-to-video.json) (t2v), [two-stage.json](../workflows/templates/ltx2/two-stage.json) (base -> latent upsample -> refine), diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 56fb1296..6105d4ea 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -7,6 +7,143 @@ notes from commits at tag time (see below). This section is a scratch pad for items a branch's author wants the next release note to name; clear it when a release ships. +### 0.10.0 + + + +A large range: the server runs one worker per GPU, the task library grows +by a dozen finishing, face, window and music-video commands, MiniMax H3 +gains held audio, guide clips and refinement, Kandinsky 6.0 is onboarded, +and the task registry owns argument domains and coercion. One upgrade-order +note: restart the server after `update_diffusers`, and Kandinsky 6 needs +diffusers from git `main`. + +**Multi-GPU servers** (docs/WORKER_GUIDE.md, docs/SERVER.md) + +- `--devices` starts a persistent worker per card; a VRAM-aware dispatcher + with backfill and cache affinity routes each job, and a declared + `vram_estimate` is admitted against the pool's largest card (#675, #676, + #677, #685). +- `get_health` and `get_memory` name the workers per card; `get_memory` and + `clear_memory` take a `device`; a job records its device ordinal and card + in `get_job`, `list_jobs` and the UI (#677, #678, #682, #777). +- A cancel sent between dispatch and the worker's start lands, and a + cancelled queued job leaves no bookkeeping behind. +- `get_memory` splits host RSS into anonymous and file-mapped pages (#709). + +**New tasks** (docs/TASKS.md) + +- Finishing: `grade` tonal controls, `sharpen`, `film_grain`, `apply_lut` + from a strictly parsed `.cube` (uploadable, listed as kind `lut`) or an + in-memory palette, with alpha kept through every one (#633, #634, #635, + #636, #775). +- Faces: `crop_face_track` follows one face and records its track; + `paste_face_track` pastes a repaired crop back, feathered (#622, #623). +- Long video: `window_video` and `join_windows` process a clip in + overlapping fixed-length windows, range-read with a uint8 join; + `fit_to_model` and `restore_to_source` take a clip to a model's grid and + back (#628, #629, #631, #632, #780). +- Music video: `analyze_beats` (onset/DP tracking with anchor calibration), + `plan_cuts` (a cut list from lyrics and beats on the render grid), + `trim_video`, `separate_stems` (htdemucs) and `ingredients_grid` (#604, + #607, #625, #626, #627). +- `check_script` compares a take's transcript to its expected lines, + shot-aware, discarding words heard over silence and Whisper repetition + loops; `templates/check-script` retires the by-eye check (#643, #644, + #645). +- `transcribe_audio` resumes after a mid-clip stop and trims word bounds to + waveform energy; `attribute_voices` handles zero-length and split lines + (#617, #661, #672). + +**MiniMax H3** + +- `hold_audio` generates video to a held soundtrack; the music-video and + `match_audio` chains hold their track instead of referencing it (#618, + #619). +- `guides`: clips held as condition rows on t2va/fl2va, with `"audio": true` + and guide continuity across chains; guides are charged in the VRAM + projection (#648, #649, #650, #779). +- A refine schedule block with `refine_strength`, its sigma grid and VAE + chunk sizes pinned to diffusers (#620, #770). Width and height declare + their multiple-of-32 constraint (#789). +- The dialogue-probe audit (docs/audit) refuted the five H3 dialogue rules + it tested (#640). + +**LTX-2.5** + +- `LTX2RefinePipeline` and `templates/ltx2/refine-in-place` with lem-tuned + strength ladders; `face-repair` and `restore-long` templates; both 2x + templates fit and restore (#624, #630, #632, #638, #639). + +**Kandinsky 6.0** (#663) + +- `templates/kandinsky6/` text-to-video, image-to-video and + generate-and-upscale (Lite-distill + tiled SR, 24GB), the `kandinsky-6` + plugin skill. Needs diffusers from git `main` (`Kandinsky6*` is in no + release yet); not on Apple Silicon (the VAE decode keeps MPS memory per + tile). +- `update_diffusers` / `POST /api/system/diffusers/update` now really + replaces a git install whose dev version string is unchanged; install.sh + and install.ps1 do the same. Restart the server after an update: admission + still checks class names against the server process's old import. +- A completed model download on huggingface_hub 1.33 is no longer reported + `failed`. + +**Validation and the task registry** + +- `@register_command` declares a task's argument domains, choices, static + checks and media arguments; `whole_number` and `real_number` are the one + numeric coercion in `dw/tasks/`, so a value is refused at validate and at + run or at neither (#773, #774, #775). +- Literal `ingredients_grid`, `select`, `film_grain`, `join_windows`, + `apply_lut`, `plan_cuts` and `check_script` arguments are refused at + validate at their argument path (#639, #646, #657, #785). +- `validate_workflow` names an unpriced composition's children, and a + `per_entry` list's index fields no longer make its estimate unknown + (#655, #772). + +**Security** + +- Media and remote-encoder requests from a workflow are dialed at the + address the host policy checked (DNS rebinding closed), every redirect is + re-validated, a POST follows the same path as a GET, and the HuggingFace + token is dropped on a redirect that leaves the host or downgrades to + http (#683, #686, #690). +- Untrusted (the default), a URL whose host does not resolve or is + percent-encoded is refused at validation rather than left to the fetch; + `--trust-workflows` keeps the old behaviour. A `file://` URL is refused as + a media location at validate and at the loader (#618). +- A fetched body is capped at 1 GiB and a fetch at 10 minutes end to end; + a chunked upload is counted as it streams (#689). Use `upload_asset` for + larger media. +- No response carries the server's absolute paths: job warnings, log lines + and phase details name an input by its `asset:` or `output:` reference, + `get_job` no longer returns a history row's `spec`, and the validate + plan's `output_dir` is gone - `plan.workspace` names the same place. + +**Fixes** + +- Chains: seams record their trim and realized crossfade, a saved + segment-backed video hands downstream its frames, a bare string for chain + prompts is refused, and the segment label clears on a failed run (#651, + #653, #660, #662, #667). +- Shots: a `for_each` member's inner shot names are qualified, a cut inside + a shot drops the seam attributes that opened it, and a join that saves + nothing still names its shots (#670, #674, #680). +- Audio: a shot consumed by `match_levels` draws no headroom warning, + `resample_waveform` returns exactly the rounded count, `video_frames` + keeps the source fps, and the music-video soundtrack is built from the + song pieces under each cut (#671, #673, #716, #788). +- uint8 frames are no longer colour-inverted by `export_to_video` (#679); + `get_job_workflow` folds recorded arguments on an unrealized job (#647); + lazy diffusers type resolution is serialized across threads (#787). + +**Plugin** + +- `kandinsky-6` skill; `minimax-h3` gains `references/dialogue.md`, the + guides and music-video render rules; `series-episodes` takes an optional + look step (palette, `apply_lut`, `film_grain`) (#637, #642, #665, #762). + ### 0.9.0 diff --git a/docs/SECURITY.md b/docs/SECURITY.md index fd1ee783..16db319b 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -143,7 +143,20 @@ One policy now answers all of it, untrusted: rather than one of them three seconds into a queued job (#124). The remedy for a file outside is to put it in the asset library and use an `asset:` reference. Containment is checked **before** existence, so the refusal - cannot be used as a file-existence oracle. + cannot be used as a file-existence oracle. Validation finds a path by its + argument's name (`image`, `*_video`, `location`, ...); a task argument + that reads a file under a generic name, like `join_windows`' `source`, is + declared on its registration (`@register_command(media_arguments=...)`, + read through `TASK_MEDIA_ARGUMENTS`, #692) so it is refused at the same moment + (#630), as are the finishing tasks' `media`, `apply_lut`'s `lut` (#635), + and the face-track and fit tasks' `clip`, `repaired`, `track` and `fit` + (#773). `lut`, `track` and `fit` are only ever read from a file on the + server, so an http(s) URL there is refused as a URL + (`LOCAL_ONLY_TASK_ARGUMENTS`). +- **A URL with any other scheme** (`file://`, `s3://`, ...) is refused at + validation and at the loader, whatever the trust posture. No loader opens + one, but as a relative path it joined onto the workflow directory and + passed containment, so validation called it clean (#618). - **A glob** is contained the same way, on the fixed directory its pattern starts from, and every match is re-checked on its real path so a symlink cannot carry the expansion out. @@ -155,17 +168,33 @@ One policy now answers all of it, untrusted: **Tailscale's tailnet range** (and Alibaba's metadata address): a workflow that fetches media from another machine on your tailnet is refused unless it runs under `--trust-workflows`. -- **Every redirect is re-checked.** A media fetch (`safe_get`) never lets the +- **Every redirect is re-checked.** A media fetch (`safe_get` in `dw/outbound.py`) never lets the HTTP client follow a redirect on its own: it follows at most 5 hops (`MAX_MEDIA_REDIRECTS`), and each `Location` passes the same scheme and host policy before it is dialed. A public URL answering `302` to `http://127.0.0.1:8765/api/server` is refused with the target named, and nothing is fetched from it. Images are decoded from the fetched bytes and still go through diffusers' `load_image`, so EXIF orientation and RGB - conversion are unchanged. + conversion are unchanged. Each hop's host is resolved once, by the policy, + and dialed at the address it checked - so a name that answers differently + to the lookup and the connect (DNS rebinding) reaches only what was + checked - and refused when it does not resolve or is percent-encoded (the + host is read with the HTTP client's own parser, so `127%2e0%2e0%2e1` is + 127.0.0.1 to the policy too). An untrusted fetch ignores proxy and + `~/.netrc` settings from the environment. The body is capped at + `MAX_MEDIA_BYTES` (1 GiB) and the whole fetch at `MEDIA_TOTAL_TIMEOUT` + (600 s) - the connection is closed at the deadline, so a server + trickling its response headers or its body cannot hold the worker; a 303 (or a + 301/302 to a POST) is followed as a GET, as requests itself does; and + `remote_text_encoder` POSTs through the same path with a timeout. Two limits + to know: a trusted run behind a SOCKS proxy is not deadline-bounded (SOCKS + brings its own connection classes), and the TLS handshake is bounded by the + per-operation timeout only. - **`remote_text_encoder.url`** is https-only, and the HuggingFace token is attached only for `huggingface.co`, `huggingface.cloud` and `hf.space`. An endpoint elsewhere is still reachable; it just does not get the credential. + A redirect that leaves the host, or drops from https to http, is followed + without the token. - **`model_name`** must be a Hub repo id or a path inside a root - the same shape check `download_model` has always applied to `repo_id`. A URL is neither, and is refused as such rather than resolving into the workflow's @@ -269,5 +298,5 @@ SecurityError ## Testing ```bash -pytest tests/test_security.py tests/test_locations.py tests/test_workflow_trust.py -v +pytest tests/test_security.py tests/test_locations.py tests/test_outbound.py tests/test_workflow_trust.py -v ``` diff --git a/docs/SERVER.md b/docs/SERVER.md index da763f29..90c83c8f 100644 --- a/docs/SERVER.md +++ b/docs/SERVER.md @@ -16,6 +16,8 @@ python -m dw.serve --workspace ~/studio python -m dw.serve --workspace ~/studio --examples-dir ~/src/diffusers-workflow/workflows python -m dw.serve --host 0.0.0.0 --token "some-long-random-string" # reachable off this machine python -m dw.serve --host 0.0.0.0 --token "..." --mcp # ...and drivable by an agent on another machine +python -m dw.serve --devices cuda:1 # run the worker on the second card +python -m dw.serve --devices cuda:0,cuda:1 # one worker per card, two jobs at once python -m dw.serve --trust-workflows # only if nothing untrusted can reach POST /api/jobs - see Security model ``` @@ -26,6 +28,26 @@ The server keeps a persistent GPU worker underneath: models stay loaded between runs, so re-running a workflow with a new prompt skips the load entirely. +### Choosing the card + +`--devices cuda:1` runs the worker on that card. The worker is pinned to it +with `CUDA_VISIBLE_DEVICES` (and `DW_DEVICE=cuda`), so inside the worker the +card is the only one it can see; a bare `cuda`, or no flag at all, pins +nothing and behaves as before. The `devices` setting is the standing form of +the same choice, and the flag wins over it. A list (`--devices cuda:0,cuda:1`) +runs one worker per card, each running one job at a time. A card named twice, +or a bare `cuda` in a list of several, is refused at startup, as is a card the +machine lacks, with the message naming the cards that are present +(`cuda:0 (NVIDIA GeForce RTX 4090)`, ...). + +Each job's record carries `device` - the card it ran on, as the server +addresses it (`"cuda:1 NVIDIA GeForce RTX 3090"`), set when the job starts +running and null for a job still queued and for jobs recorded before the +field existed. Observed cost figures are bucketed by it: a figure counts the +runs from cards of the same name as the one the server runs on now, so a +3090's runs never inform a 4090's estimate. Rows from before jobs carried a +`device` count only while the server is on the box's default card. + ## The pages The UI is organised by workspace. A sidebar lists every workspace on the @@ -165,7 +187,7 @@ agent from another machine, plus the queue across every workspace: | Route | What it does | | --- | --- | | `POST /api/jobs` | Queue a run: `{"workflow_path": ...}` or an inline `{"workflow": {...}, "base_dir": ...}`, plus `arguments` for variable overrides. `workflow_path` accepts a stored workflow name as listed by `/api/workflows` (with or without `.json`, nested names included), or a relative/absolute path that still resolves under `--workflow-dir` - confined the same way the `/api/workflows` CRUD routes are; a path that names a real file outside that directory is rejected with 400, not opened. Answers with argument warnings from signature checking. Takes an optional `acknowledged_cost`: `true` is recorded as `acknowledged: boolean`; the object `{fingerprint, minutes, downloads}` from a validate answer's `plan` is `bound` - the server re-plans the run for the arguments given and answers **409** when the fingerprint differs or a repo in `downloads_required` is not in `downloads` (a download that has since vanished is not a refusal); the body is `{"detail": {message, reason: "fingerprint" \| "downloads" \| "unplannable", acknowledged, plan, acknowledge}}` with the current plan, so the caller re-quotes from it, and `acknowledge` - the `{fingerprint, minutes, downloads}` to resend - whenever there is a plan. A `null` in `downloads` (a URL with no repo) is ignored. `minutes` is recorded, never compared. Nothing is required: the web UI and every caller that sends nothing are `acknowledged: none`, and every job answer and history row carries `acknowledged` (and `acknowledged_cost` when bound). `POST /api/jobs/{id}/rerun` takes the same field and checks against the stored spec; a fresh seed does not change a fingerprint. | -| `GET /api/jobs?workspace=&status=&limit=` | Queue + history summaries, oldest first, with `total` beside them. `status` narrows to one state or a comma-separated set (`queued`, `running`, `succeeded`, `failed`, `cancelled`; anything else is a 400); `limit` keeps the newest N, and `total` still reports how many matched, so a bounded answer cannot be mistaken for a complete one. No parameters means every job, which is what the web UI polls | +| `GET /api/jobs?workspace=&status=&limit=` | Queue + history summaries, oldest first, with `total` beside them. `status` narrows to one state or a comma-separated set (`queued`, `running`, `succeeded`, `failed`, `cancelled`; anything else is a 400); `limit` keeps the newest N, and `total` still reports how many matched, so a bounded answer cannot be mistaken for a complete one. Each summary carries `device`, the card the job ran on (`"cuda:1 NVIDIA GeForce RTX 3090"`; null while queued and for older jobs). No parameters means every job, which is what the web UI polls | | `GET /api/jobs/{id}` | Full detail: spec, events, manifest, error. A manifest entry for a step served from the step cache carries `reused: true`. Every entry carries `subfolder` - the in-run subfolder the step's `result.subfolder` chose, `''` for none. A `for_each` step appears in the manifest as its members (`shot@wide_open`, `shot@closeup`), because the manifest records what ran; the run's `workflow.json` keeps the `for_each` form, because it records what was asked Carries `output_kinds`: each manifest file mapped to its media kind (`image`, `video`, `audio`, `text`, or null), and each file-carrying run event (`step_end`) carries the same map for its own files. | | `DELETE /api/jobs/{id}/run` | Delete the run directory a finished job wrote, whole, from the output root the job ran against (no workspace selector applies): `{job_id, run_dir, deleted, run_swept}`. 404 for an unknown job or one with no run directory, 409 for one still queued or running | | `GET /api/jobs/{id}/workflow` | The workflow the job ran: `{id, definition, realized, seed_variable}`. `seed_variable` names the variable a `new_seed` rerun would draw into (null when the workflow has none), read from the workflow as written rather than the realized copy, whose seed is pinned. `realized: true` is the copy the run itself wrote (`workflow.json` in its run directory), with arguments, seed, prompts and `output:latest` pinned; `false` falls back to the submitted definition, which is what a job from before run tracking has. 404 means neither is readable - the job itself still is. The equivalent MCP tool is `get_job_workflow` (see [MCP.md](MCP.md#diagnose)) | @@ -177,8 +199,34 @@ agent from another machine, plus the queue across every workspace: | `POST /api/jobs/{id}/rerun` | Re-queue a finished job's spec. Body `{"new_seed": true}` draws a fresh seed into the workflow's seed variable instead of repeating the original arguments; 400 when the workflow pins its seed to a literal or names none. A plain rerun of a seeded workflow is served whole from the step cache — the earlier run's files, republished with `reused: true`, generating nothing | | `POST /api/jobs/{id}/move` | Reorder a queued job: `{"direction": "up"\|"down"\|"front"\|"back"}`. Job listings carry each waiting job's `queue_position`. | -One job runs at a time (it is one GPU); submissions queue in order, and -the waiting portion of the queue can be reordered. +Each worker runs one job at a time, so with one device (the default) one job +runs at a time; submissions queue in order, and the waiting portion of the +queue can be reordered. With several `--devices` the dispatcher walks the +queue in order and gives each job the first free card, in `--devices` order, +that fits it. A job that fits no free card keeps its place while a smaller +one behind it takes the card (backfill). + +What a job needs: the workflow's declared `vram_estimate` projected over the +job's arguments (hard: a job bigger than every card is refused at submit with +a 400 naming the largest card); else the smallest `cost` entry's `vram_gb` for +this device type (soft: if no card is that big, the job runs on any card, as +on one card); else any card. A card's size is its GiB rounded up (a 3090 +counts as 24), which a `cost` figure is compared to; a declared estimate is +held to the card's GiB to one decimal (a 3090's 23.6), the same ceiling +admission checks it against. Cancel reaches only the worker running that job, and a worker +crash fails only its own job. The worker started later gets a higher Linux +`oom_score_adj` (+100 over the highest other live worker), so a host-RAM +squeeze kills the later job; nothing is written with one worker. There is no +host-RAM gate at dispatch: two big jobs can still exhaust host RAM together. +Among the free cards a job fits, dispatch prefers the one it has an affinity +for: a rerun goes to the card that ran the original, and any job to the card +whose worker last ran the same workflow (the same file, or an inline +definition's `id`) - that worker's pipelines and step cache are warm, and a +worker frees both when the workflow changes. A busy preferred card is not +waited for. A validation's cache probe and its `plan.estimate` ask about the +card the job would be dispatched to now, and the estimate names it in +`priced_for`. Memory is per card: `GET /api/memory` and `POST +/api/memory/clear` take an optional `device`. ### Progress events @@ -545,9 +593,11 @@ The editor's forms come from these; they are just as usable from scripts: caller waits through for nothing. Everything else - `.json`, `.md`, `.txt`, an unrecognized extension - deflates; so does the export zip's text files (`workflow.json`, `manifest.json`, `job.json`, the README) -- `POST /api/uploads?filename=...` — the raw bytes of one image, video or audio file +- `POST /api/uploads?filename=...` — the raw bytes of one image, video or audio file, + or a `.cube` 3D LUT for `apply_lut` (200MB ceiling, checked from `Content-Length` before a byte is read, and - again on the body; extension held to the allowed image/video list), saved + again on the body; extension held to the allowed image, video, audio and + `.cube` list), saved into the asset library's `uploads/` subfolder - the shared library at `/common/assets` when `shared=true`, this workspace's own otherwise - under a generated name, or under `asset_name` when one is given (`cast/priya-voice.wav`, folders allowed, @@ -569,7 +619,10 @@ The editor's forms come from these; they are just as usable from scripts: installed diffusers version/commit, and a background diffusers install/ update (refused while a job is running or queued). The POST body is optional JSON, `{"commit": ..., "revert": ...}`: with neither, it - `pip install --upgrade`s from GitHub HEAD; `commit` (7-40 hex characters, + installs from GitHub HEAD - forced in with `--force-reinstall --no-deps`, + since main keeps one dev version string for a release cycle and a plain + upgrade would keep an older commit, then a plain `pip install diffusers` + for any dependency the new commit adds; `commit` (7-40 hex characters, validated before it reaches the command line) pins the git install to that commit instead of HEAD; `revert: true` pins back to the known-good published release instead of installing from git - the diffusers floor @@ -583,14 +636,23 @@ The editor's forms come from these; they are just as usable from scripts: `reason` (`job_running`, `worker_stopped`, `worker_busy`, `worker_unreachable`) and `age_seconds`, so a cached reading is never mistaken for the worker's memory now - `info: null` means nothing has been - measured because nothing is resident. health also reports `hostname`, + measured because nothing is resident. Without `device` (`?device=cuda:1`) + the first card's reading is at the top level and `workers` has one entry per + card, each naming its `device`; with it, that card's alone (400 for a card + this server has no worker on). health also reports `hostname`, `device` and whether `mcp` is mounted, so a remote client can tell which - machine answered + machine answered. `current_job` is the longest-running job and `worker_alive` + is true if any worker is alive; `workers` has one entry per card (one with a + single device): `{device, name, vram_gb, current_job, alive, + host_memory_rss_mb}` - `device` is the ordinal (`cuda:1`) and `name` the + GPU's own name, the two halves of a job's `device` - the last absent when the worker process isn't running - `POST /api/memory/clear` (#221) — drops every loaded pipeline and the step - cache (MCP `clear_memory`), and returns the - memory reading taken right after. Refused with 409 while a job is running - or queued - the queue is FIFO, so the caller retries once it finishes - rather than this call blocking until it does + cache on each idle card (MCP `clear_memory`), and returns the memory + reading taken right after. A card running a job is left alone: `workers` + reports it `{device, cleared: false, reason: "job_running", job}`. With + `?device=cuda:1` it clears that card alone and answers `{cleared, info, + device}`. Refused with 409 when no card asked about is idle - the caller + retries once the job finishes rather than this call blocking until it does - `GET /api/server` — connection details for the Server page: `hostname`, `version`, `device`, the `bind_host`/`port`/`wildcard_bind` the server was started with, `auth_required` (whether a token is configured - never the diff --git a/docs/TASKS.md b/docs/TASKS.md index 064cccc5..c1911661 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -1,6 +1,6 @@ # Task Commands -Tasks are utility operations that run outside of pipeline inference. Use them for image preprocessing, data gathering, and other non-model operations. +Tasks are utility operations that run outside of a pipeline step. Use them for preprocessing, assembly, data gathering, and analysis - most are plain code, and some run a small helper model (an upscaler, a face detector, a stem separator). ```json { @@ -36,6 +36,75 @@ reported as clean successes. The commands refuse the same values at run time, which is what catches one that arrived from a `variable:` or an earlier step rather than being written in the file. +A numeric argument is read the same way by every command and by validation +(`whole_number` and `real_number` in `dw/task_domains.py`, the only numeric +coercion in `dw/tasks/`). A number may arrive as a JSON number or as a numeric +string - a `variable:` resolved from the command line is text - so `3`, `3.0`, +`"3"` and `"3.0"` are all the whole number 3, and `"23.976"` is a frame rate. +A whole-number argument (a frame count, an index, a sample rate, a pixel size) +refuses a fractional value such as `3.5` or `"22050.5"` rather than truncating +it, naming the argument and saying "whole number". Every numeric argument +refuses `true`/`false`, text that is no number (`"abc"`) and an infinite or NaN +value. What validation refuses, the run refuses with the same sentence, and +the reverse. +`Task.run` reads every declared numeric argument before the command sees it +(`coerce_arguments`), so a handler is handed a number, never the string it +arrived as: `exposure: "0.5"` grades exactly as `0.5` does. An argument whose +only rule is being a number, such as `grade`'s `exposure` or a gain in dB, +declares the `finite` domain, so validation reads it too. + +## Adding a task + +A task is one function, registered with `@register_command` in +`dw/tasks/registry.py`. Its signature is its argument schema, and everything +validation knows about it beyond the signature is declared on the same +decorator, so a new task cannot be half-registered (#692): + +```python +from .registry import register_command +from ..task_domains import NON_NEGATIVE, POSITIVE, my_task_errors + + +@register_command( + "my_task", + domains={"num_frames": POSITIVE, "start_frame": NON_NEGATIVE}, + choices={"mode": ("fast", "best")}, + static_check=my_task_errors, + media_arguments=("source",), +) +def my_task(source, num_frames, start_frame=0, mode="fast"): ... +``` + +- `domains` - each numeric argument's domain (`dw/task_domains.py`'s + `POSITIVE`, `NON_NEGATIVE`, ...): refused at validation at + `steps[i].task.arguments.`, and again at run time when the task calls + `check_arguments`. +- `choices` - each argument's literal choices; `get_task` lists them. +- `static_check` - a function of the step's `arguments` that returns + `(argument, message)` pairs for rules across arguments. It lives in + `dw/task_domains.py`, named `*_errors`, and is registered to exactly one + command - or, when it needs the task's own parser, in the task's module + behind an import on use, as `attribute_voices`' does. +- `whole_numbers` - the numeric arguments that take only a whole number; the + rest of `domains` take any real number. Validation refuses `3.5` for one + of these, as the task's own `whole_number` call does at run time. +- `media_arguments` - an argument that reads a file under a name that does not + say so (`source`, `media`, `lut`, `clip`, `track`). Validation confines it as it does `image` + or `*_video`; an argument left out of it is not path-checked. + +`TASK_ARGUMENT_DOMAINS`, `TASK_ARGUMENT_CHOICES`, `TASK_STATIC_CHECKS`, +`TASK_WHOLE_NUMBER_ARGUMENTS` and `TASK_MEDIA_ARGUMENTS` are read-only views of these declarations; +`tests/test_task_registry_rules.py` pins them to the registry and every name in +them to the command's signature. + +A new task is: + +1. a module in `dw/tasks/` holding the function and its decorator; +2. one import line for that module in `dw/tasks/task.py` - the registry is + filled by importing `dw.tasks.task`, and modules are not discovered + automatically, so a module it does not import registers nothing; +3. a section in this file. + ## Image Processing ### ControlNet Preprocessors @@ -177,6 +246,264 @@ The `resize_bucket` command snaps an image to the closest model-native aspect ra For example, a 1600x900 photo (16:9) at resolution 1024 becomes 1792x1024. A 800x600 photo (4:3) becomes 1344x1024. +### grade + +`grade` is the finishing pass: exposure, contrast, tonal range, local contrast, +white balance, colour and a matte or vignette look, on CPU, for an image or a +video. A video is graded one frame at a time and keeps its frame count, frame +rate and audio. Unlike the other image commands, its input is named `media`, +and a file path or `asset:`/`output:` reference to a video is accepted +directly. An alpha channel passes through untouched. + +```json +{ + "task": { + "command": "grade", + "arguments": { + "media": "previous_result:generate_video", + "highlights": -0.3, + "shadows": 0.2, + "clarity": 0.25, + "fade": 0.15, + "vignette": 0.3 + } + }, + "result": { "content_type": "video/mp4" } +} +``` + +Every argument is optional and defaults to identity, so a step with none +returns the input pixels. The step's job events carry one "applied" log naming +every argument that is not at its identity value. + +| Argument | Range | Identity | Effect | +| -------- | ----- | -------- | ------ | +| `exposure` | any | 0.0 | Stops to brighten (positive) or darken (negative): a multiply by 2^exposure | +| `contrast` | 0 or above | 1.0 | Multiplier around mid grey; below 1 flattens | +| `whites` | -1.0..1.0 | 0.0 | Moves the white point: the top of the curve up or down, black held | +| `blacks` | -1.0..1.0 | 0.0 | Moves the black point: the bottom of the curve up (lift) or down (crush), white held | +| `highlights` | -1.0..1.0 | 0.0 | Lifts or pulls down tones above mid grey, by a smooth luma mask; tones at and below mid grey are untouched | +| `shadows` | -1.0..1.0 | 0.0 | The same for tones below mid grey; tones at and above mid grey are untouched | +| `clarity` | -1.0..1.0 | 0.0 | Adds (positive) or removes (negative) medium-detail local contrast in the midtones | +| `temperature` | -1.0..1.0 | 0.0 | Warmer (toward red) or cooler (toward blue) | +| `tint` | -1.0..1.0 | 0.0 | Toward magenta (positive) or green (negative) | +| `saturation` | 0 or above | 1.0 | Multiplier around each pixel's luma; 0 is greyscale | +| `fade` | 0.0..1.0 | 0.0 | Lifts the black floor and flattens the shadows toward it, white kept: a matte look | +| `vignette` | -1.0..1.0 | 0.0 | Darkens (positive) or lightens (negative) toward the corners; the centre is untouched | + +**Order of operations:** exposure, contrast, whites/blacks, +highlights/shadows, clarity, temperature/tint, saturation, fade, vignette. The +order is fixed whatever order the arguments are written in: fade comes after +the tonal controls so `blacks` cannot pull its floor back down, and vignette is +last so it darkens the finished picture. + +A value outside its range is refused by `validate_workflow`, naming the +argument and the range, and again at run time when it arrives from a +`variable:` or an earlier step. `get_task("grade")` reports each range. + +### sharpen + +`sharpen` is an unsharp mask, on CPU, for an image or a video: the image is +blurred with a Gaussian, and the difference between the image and its blur (the +detail) is added back, scaled by `amount`. Like `grade`, its input is named +`media`, and a file path or `asset:`/`output:` reference to a video is read with +its audio. A video is sharpened one frame at a time and keeps its frame count, +frame rate and audio. An alpha channel passes through untouched. + +```json +{ + "task": { + "command": "sharpen", + "arguments": { + "media": "previous_result:generate_video", + "amount": 0.8, + "radius": 1.5, + "threshold": 3 + } + }, + "result": { "content_type": "video/mp4" } +} +``` + +| Argument | Range | Default | Effect | +| -------- | ----- | ------- | ------ | +| `amount` | 0 or above | 1.0 | How much of the detail is added back; 0 is identity. Applied in whole percent, so it is rounded to 0.01 | +| `radius` | above zero | 2.0 | The Gaussian blur's radius in pixels: the scale of the detail that is sharpened | +| `threshold` | 0..255 | 0 | The smallest difference, in channel levels, between a pixel and its blur that gets sharpened; smaller differences are left alone. A whole number of levels: a fractional value is rounded | + +A value outside its range is refused by `validate_workflow`, naming the +argument and the range, and again at run time when it arrives from a +`variable:` or an earlier step. `get_task("sharpen")` reports each range. + +### film_grain + +`film_grain` adds seeded film grain, on CPU, to an image or a video. Its input +is `media`, as in `grade` and `sharpen`: a video is processed one frame at a +time and keeps its frame count, frame rate and audio, and an alpha channel +passes through untouched. The grain is Gaussian noise, strongest in the +midtones and weaker toward black and white. + +```json +{ + "task": { + "command": "film_grain", + "arguments": { + "media": "previous_result:generate_video", + "amount": 0.15, + "size": 1.5, + "chroma": 0.2 + } + }, + "result": { "content_type": "video/mp4" } +} +``` + +| Argument | Range | Default | Effect | +| -------- | ----- | ------- | ------ | +| `amount` | 0.0..1.0 | 0.1 | Grain strength; 0 is identity | +| `size` | 1 or above | 1.0 | Grain size in pixels: the noise is generated at 1/size resolution and upsampled | +| `chroma` | 0.0..1.0 | 0.0 | 0 puts the same grain on every channel (brightness only, hue unchanged); 1 draws independent grain per channel; values between mix the two | +| `seed` | a whole number, 0 or above | the workflow's or step's seed | Seeds the grain | + +**Seeding:** one generator is made per step from the seed and consumed frame by +frame, so every frame of a video gets different grain and the same seed +reproduces the whole output byte for byte. A workflow run always has a seed (a +random one is drawn when the workflow names none, and recorded in the run's +manifest), so rerunning a job reproduces its grain. An explicit `seed` +argument overrides the workflow's or step's seed. The step's job events carry +a log naming the seed used. + +A value outside its range is refused by `validate_workflow`, naming the +argument and the range, and again at run time when it arrives from a +`variable:` or an earlier step. `get_task("film_grain")` reports each range. + +### apply_lut + +`apply_lut` colours an image or a video through a 3D lookup table (LUT), read +from a `.cube` file or built from a `palette`, on CPU. Its input is `media`, as in `grade`: a video is +processed one frame at a time and keeps its frame count, frame rate and audio, +and an alpha channel passes through untouched. Each pixel's colour is looked up +with trilinear interpolation, and the result is blended with the original by +`strength`. + +```json +{ + "task": { + "command": "apply_lut", + "arguments": { + "media": "previous_result:generate_video", + "lut": "asset:looks/teal-orange.cube", + "strength": 0.7 + } + }, + "result": { "content_type": "video/mp4" } +} +``` + +| Argument | Range | Default | Effect | +| -------- | ----- | ------- | ------ | +| `lut` | a `.cube` file | none | The LUT: an `asset:` reference (upload one with `upload_asset`, or `POST /api/uploads`), an `output:` reference, or a path inside the workflow's directory, the asset libraries or the output root | +| `palette` | 2..16 `#rrggbb` colours | none | Instead of a `.cube`, a look built from colours ordered dark to light (below) | +| `strength` | 0.0..1.0 | 1.0 | 0 returns the original, 1 the full LUT result; values between blend the two | + +Exactly one of `lut` or `palette` is given; both or neither refuses, naming +the two. + +**A palette.** `palette` is a list of 2 to 16 `#rrggbb` colours, ordered dark +to light, spaced evenly along the luminance axis: the first colour sits at +black, the last at white. Each pixel keeps its own luminance (Rec. 709 luma) +and takes its hue and chroma from the palette at that luminance, interpolated +between neighbouring colours, so shadows lean toward the first colour and +highlights toward the last. Near black and white the chroma shrinks as far as +it must to stay in range. The 33³ table is built in memory, never written to +disk, and applied exactly as a `.cube` is (same interpolation, same `strength` +blend). The same palette always gives the same result, so one palette passed +as a `variable:` to every shot of a series gives them one look: + +```json +{ + "variables": { "look": ["#102030", "#e0c090"] }, + "steps": [ + { + "name": "look", + "task": { + "command": "apply_lut", + "arguments": { + "media": "previous_result:generate_video", + "palette": "variable:look", + "strength": 0.6 + } + }, + "result": { "content_type": "video/mp4" } + } + ] +} +``` + +A palette that is not a list, a colour that is not `#rrggbb`, or fewer than 2 +or more than 16 colours refuses, naming the bad entry - at `validate_workflow`, +and again at run time for a palette that arrives from an earlier step. + +**Where the file may be.** A literal path is held to the same roots as any +media argument, and must end in `.cube`; anything else is refused, naming only +what the workflow wrote. The file is read and parsed once per step, not once +per frame. + +**The `.cube` the parser takes** is the 3D subset of the format, read strictly. +Anything else refuses the step, naming the file and the line: + +- at most 16 MiB of UTF-8 text; +- only `TITLE`, `LUT_3D_SIZE`, `DOMAIN_MIN`, `DOMAIN_MAX`, `#` comment lines, + blank lines and data rows. `LUT_1D_SIZE` (a 1D LUT) and every other keyword + refuse; +- every keyword before the data, and `LUT_3D_SIZE` exactly once, 2..65; +- a domain of exactly `0 0 0` to `1 1 1` (the default when the file names + none); +- exactly size³ data rows of three finite numbers, each in 0..1, listed red + fastest, then green, then blue, as the format specifies. + +Some grading applications write values slightly outside 0..1 or a wider domain; +those files are refused rather than clipped. `strength` outside 0..1 is refused +by `validate_workflow` and again at run time when it arrives from a `variable:` +or an earlier step. + +### A finishing chain + +The three finishing commands chain through `previous_result:`, each taking the +last one's output as its `media`. Grade first, sharpen the graded picture, and +add grain last so it is not itself sharpened: + +```json +{ + "steps": [ + { + "name": "graded", + "task": { + "command": "grade", + "arguments": { "media": "previous_result:generate_video", "fade": 0.15, "vignette": 0.3 } + }, + "result": { "content_type": "video/mp4" } + }, + { + "name": "sharpened", + "task": { + "command": "sharpen", + "arguments": { "media": "previous_result:graded", "amount": 0.6 } + }, + "result": { "content_type": "video/mp4" } + }, + { + "name": "grainy", + "task": { + "command": "film_grain", + "arguments": { "media": "previous_result:sharpened", "amount": 0.12, "size": 1.5 } + }, + "result": { "content_type": "video/mp4" } + } + ] +} +``` + ## Video Processing | Command | Description | Extra Arguments | @@ -184,6 +511,10 @@ For example, a 1600x900 photo (16:9) at resolution 1024 becomes 1792x1024. A 800 | `get_first_frame` | Extract first video frame | | | `get_last_frame` | Extract last video frame | | | `get_frame` | Extract frame at index | `frame_index` | +| `window_video` | One overlapping, fixed-length window of a long video, with its audio | `index`, `num_frames`, `overlap`, `fps` - see [window_video](#window_video) | +| `join_windows` | Blend processed windows back into one video the source's length, with the source's audio | `videos`, `source`, `num_frames`, `overlap`, `curve`, `fps` - see [join_windows](#join_windows) | +| `fit_to_model` | Fit a video into a model's working size and exact frame count; returns `{video, fit}` | `width`, `height`, `num_frames`, `mode` (`letterbox`, `stretch`, `crop`), `downscale` - see [fit_to_model and restore_to_source](#fit_to_model-and-restore_to_source) | +| `restore_to_source` | Put a model's output back at its source's size and length, from the `fit` record | `fit` - see [fit_to_model and restore_to_source](#fit_to_model-and-restore_to_source) | The frame commands accept videos in any shape a result carries them: PIL frame lists, numpy or torch frame arrays, and audio+video pairs (LTX-2, MiniMax H3). @@ -480,6 +811,393 @@ assembly templates once did and what made their output visibly wider than the source. The join tasks refuse shots of different sizes, so no normalization step is needed before them. +### trim_video + +Keep one span of a video's frames, and its audio over the same span: + +```json +{ + "task": { + "command": "trim_video", + "arguments": { + "video": "previous_result:shot", + "start_frame": 12, + "num_frames": 72 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `video` | Yes | The clip. A path, `asset:` or `output:` reference is read with its audio; an earlier step's video is used as it is. A URL is fetched as frames only | +| `start_frame` | Yes | The first frame kept, from `0` | +| `num_frames` | Yes | How many frames are kept: `1` or more | +| `fps` | No | The clip's frame rate. Defaults to the rate the clip carries; needed only to cut its audio, which is refused without one | + +Frames `[start_frame, start_frame + num_frames)` are kept. The audio is cut to +the same span at the track's own sample rate: each end is `frames_to_samples` +of a frame index, rounded on its own, so consecutive trims tile the track with +no sample lost or repeated. A clip with audio (or an audio+video pair) comes +back as an audio+video pair with its frame rate and sample rate kept; a bare +frame list comes back as frames. + +The shots the clip carries are clipped to the span and re-based to start at +`0`, with their sample side cleared. A clip that carries none (a decoded file, +a fresh render) comes back as one shot spanning the kept frames, named after +the file it was read from (else `video 1`), its samples measured off the cut +track. A span that reaches past the clip's end is refused, naming the clip's +frame count - it is never shortened or padded - and so is a `num_frames` of +`0` or less or a negative `start_frame`; `validate_workflow` catches these on +literal values. + +### window_video + +Cut one overlapping, fixed-length window out of a long video, so a +video-to-video model that reads at most one bucket of frames (121 for LTX) +can work through a longer source a window at a time: + +```json +{ + "name": "window", + "for_each": [ + {"name": "w0", "index": 0}, + {"name": "w1", "index": 1}, + {"name": "w2", "index": 2} + ], + "task": { + "command": "window_video", + "arguments": { + "video": "asset:long-take.mp4", + "index": "item:index", + "num_frames": 121, + "overlap": 16 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `video` | Yes | The long source. A path, `asset:` or `output:` reference is read with its audio; an earlier step's video is used as it is. A URL is fetched as frames only, so its window is silent | +| `index` | Yes | Which window, from `0` | +| `num_frames` | Yes | Frames per window - the model's bucket, e.g. an 8n+1 LTX length | +| `overlap` | Yes | Frames each window shares with the one before it; `0` or more, and below `num_frames` | +| `fps` | No | The source's frame rate. Defaults to the rate its file was read at; needed only to cut its audio | + +With `stride = num_frames - overlap`, window `i` covers source frames +`i*stride - overlap` up to (not including) `i*stride + stride`. The first +window starts `overlap` frames before the source and repeats its first frame +there; the last may run past the source's end and repeats its last frame. +Every window is exactly `num_frames` frames, float32 in `[0, 1]` like +`loop_frames`' output. A source of `N` frames takes `ceil(N / stride)` +windows, indexes `0` to `(N - 1) // stride`. + +The window's audio is the source's samples for its real frames, cut on the +source's own frame boundaries (`frames_to_samples`, #401), so adjacent +windows' strides tile the track exactly; the repeated frames carry silence of +the length they would have had. A source with no track gives a silent window. + +Memory (#695): a file source (a path, `asset:` or `output:`) is never decoded +whole - the window reads only the source frames it covers, from the keyframe +before them, and the soundtrack on its own, so a window of a long source +holds about one window of float32 frames plus the decoded track. An earlier +step's video is already in memory and is cut from there. + +One step makes one window: a task cannot return a list of them, so drive it +with `for_each` over `{name, index}` entries as above. It refuses an +`overlap` that is not below `num_frames`, and a negative `overlap` or `index` +- `validate_workflow` catches these on literal values - and, at run time, an +`index` whose window would start at or past the source's last frame, naming +the source's frame count and the last valid index. + +### join_windows + +The other half of `window_video`: blend the processed windows back into one +video the source's length. Give it the windows in order, the source they were +cut from, and the same `num_frames` and `overlap`: + +```json +[ + { + "name": "window", + "for_each": "variable:windows", + "task": { + "command": "window_video", + "arguments": { + "video": "variable:source_video", + "index": "item:index", + "num_frames": 121, + "overlap": 16 + } + } + }, + { + "name": "joined", + "task": { + "command": "join_windows", + "arguments": { + "videos": "gather:window", + "source": "variable:source_video", + "num_frames": 121, + "overlap": 16 + } + } + } +] +``` + +with `"windows": [{"name": "w0", "index": 0}, {"name": "w1", "index": 1}, ...]`. +A step that processes each window goes between the two, and `videos` gathers +that step instead. + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `videos` | Yes | The processed windows, in window order: `gather:` over the step that processed them | +| `source` | Yes | The video the windows were cut from, as for `window_video`'s `video`. Its frame count sets the plan and its track is the output's | +| `num_frames` | Yes | Frames per window, as given to `window_video` | +| `overlap` | Yes | Frames each window shares with the one before, as given to `window_video` | +| `curve` | No | The blend's shape across a seam: `cosine` (the default, `(1 - cos πt) / 2`), `smoothstep` (`3t² - 2t³`) or `linear` (`t`) | +| `fps` | No | The source's frame rate. Defaults to the rate its file was read at; needed only to put its audio back | + +The result has exactly the source's frame count, each source frame once: +window 0's real frames, then each later window's first `overlap` frames +blended over the previous window's last `overlap` frames, and the last +window's pad dropped. The incoming weight is `w(t)` at `t = (k + 1) / +(overlap + 1)` for seam frame `k` - the open ramp `dissolve_videos` uses, so +no seam frame is a bare copy of either side. The windows may be at a +different size from the source (a 2x upscale); the output is at the windows' +size. + +Memory (#695): the output is built as uint8, with only a seam's `overlap` +frames blended in float32, so a join holds the output at one byte a channel +plus the windows it was given - never the whole video in float32. A file +`source` is read for its frame count and soundtrack only, never its picture. + +The window count must be exactly `ceil(source_frames / (num_frames - +overlap))` - the rule has one home, `window_count` in `dw/task_domains.py`, +and `window_video`'s last valid `index` is one below it. A different count is +refused naming both numbers and the list entries (by index) to add or drop: +by `validate_workflow`, before any window renders, when the count is +knowable from the document - `source` an `asset:`/`output:` reference or a +literal path (probed header-only), `num_frames` and `overlap` literal after +substitution, `videos` a list (a `gather:` over a `for_each` step is one) - +and otherwise at run time, once the source is decoded. The validate-time +check is the task's (`dw/window_count_errors.py`), so any `join_windows` +step gets it, and it applies the same `window_count_problem` the task does. It also refuses a window whose frame count is not `num_frames` +(named by position) and windows of different sizes. An unknown `curve` and an +`overlap` that is not below `num_frames` are refused by `validate_workflow` +on literal values as well. + +The output's audio is the source's own track over exactly +`frames_to_samples(source_frames)`, and none when the source has none; the +windows' audio is discarded. Its shot records are one per window over the +frames it owns (window `i > 0` starts at its blended head), with +`overlap_frames` on every seam and cumulative `frames_to_samples` sample spans +(#401), so `assess_output` reads the seams as dissolves and its sync check +measures against the source's timeline. Like `concat_videos` and +`dissolve_videos`, its catalog shape is a cut task. + +### fit_to_model and restore_to_source + +A video-to-video model works at its own size and frame count, not the +source's. `fit_to_model` fits the source into the model's `width` x `height` +and exactly `num_frames`, the pipeline runs on the fitted video, and +`restore_to_source` puts the output back at the source's size and length: + +```json +[ + { + "name": "fit", + "task": { + "command": "fit_to_model", + "arguments": { + "video": "asset:clip.mp4", + "width": 768, + "height": 512, + "num_frames": 121, + "mode": "letterbox" + } + } + }, + {"name": "upscale", "pipeline": {"...": "reads previous_result:fit.video"}}, + { + "name": "restore", + "task": { + "command": "restore_to_source", + "arguments": { + "video": "previous_result:upscale", + "fit": "previous_result:fit.fit" + } + } + } +] +``` + +`mode` is `letterbox` (default: scale to fit, centred on black), `stretch` +(resize to fill exactly) or `crop` (scale to fill, centre-crop). The frame +count is exact: a longer source is cut to its first `num_frames` frames and a +shorter one holds its last frame. `downscale` (default 1) divides `width` and +`height`, which must both be divisible by it: a 2x upscaler whose `width` x +`height` is the output it renders takes its reference at half that, so it fits +with `downscale: 2` and passes the same `width`/`height` to the pipeline +(`templates/ltx2/upscale-clip`). The step returns `{video, fit}`, read as +`previous_result:.video` and `previous_result:.fit`. `video` is a +float32 array of frames, height, width, RGB in 0-1 that carries its `fps`, so +it goes straight to a pipeline's `video` or a reference condition's `frames`; +saved, it is written as an mp4 at that rate. The `fit` +record holds `mode`, `source_width`, `source_height`, `source_frames`, +`model_width`, `model_height`, `model_frames`, `content_box` (where the source +sits in the model frame) and `source_box` (the part of the source kept - all +of it unless `mode` is `crop`), each box `{x, y, w, h}`. + +`restore_to_source` takes the pipeline's output and that record; it has no +`scale` argument. The scale is the output's size over the fitted size and must +be the same across and down, so a 2x upscaler restores to twice the source. +Letterbox bars are cropped off, and frames are trimmed to the source's count, +dropping the held ones (an output shorter than that is returned as it is). +`crop` cannot bring back the edges it cut, so it returns the kept region at +the source's pixel density times the scale: 640x480 fitted into 512x288 +restores to 640x360. A saved video keeps its exact size; only an odd side +grows by one pixel, since the encoder needs even sides. + +Neither task carries a soundtrack. Pair the source's with `pair_audio` after +the restore. + +### crop_face_track + +Follow one face through a clip and cut a steady square around it, for a +face-detail pass that needs the face large and still: + +```json +{ + "name": "face_crops", + "task": { + "command": "crop_face_track", + "arguments": { + "clip": "asset:interview.mp4", + "crop_size": 512 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `clip` | Yes | The video - frames, an audio+video pair, or the path or URL of a video file. Named `clip` so the engine hands it over as read, with the shots it records | +| `crop_size` | No | Side of every crop in pixels, a multiple of `multiple`. Default `512` | +| `padding` | No | Space added around the face on each side, as a fraction of its size, `0` to `3`. Default `0.6` | +| `gate_full` | No | Face width over frame width at or below which strength is 1. Default `0.06` | +| `gate_zero` | No | Face width over frame width at or above which strength is 0; must exceed `gate_full`. Default `0.12` | +| `min_confidence` | No | Detections scoring below this are ignored; below `1`. Default `0.6` | +| `modulus` | No | The crop count is padded to `modulus * n + remainder` frames - the frame grid of the model the crops feed. At least `1`. Default `8` | +| `remainder` | No | The padded count's remainder, `0` up to `modulus - 1`. Default `1` | +| `multiple` | No | `crop_size` must be a multiple of this - the model's frame-size step. At least `1`. Default `32` | +| `detector_repo` | No | Hub repo holding the YuNet weights. Default `opencv/face_detection_yunet`, read at a pinned revision; any other must be a Hub repo id | +| `detector_file` | No | The detector file in that repo, a bare `.onnx` name. Default `face_detection_yunet_2023mar.onnx` | +| `device` | No | Where detection runs | + +`modulus`, `remainder` and `multiple` follow `plan_cuts`: the model's grid is +the workflow's to declare. The defaults (8, 1, 32) are LTX's 8n+1 frames and +32-pixel steps, so a workflow that names none gets the output it always did; +the `face-repair` template passes `modulus: 8, remainder: 1, multiple: 32` +explicitly. A `modulus` below 1, a `remainder` below 0 or not below `modulus`, +and a `crop_size` that is not a multiple of `multiple` are refused at +validation. + +These defaults are the task's own. The `face-repair` template tunes them on +lem for LTX-2.5 at `padding` 1.5, `gate_full` 0.03 and `gate_zero` 0.06: a +medium face (7.5% of the frame's width when tuned) already renders cleanly, and the wider +context keeps the refine from re-inventing hair and clothing. + +**Example:** [face-repair.json](../workflows/templates/ltx2/face-repair.json) — find, refine and paste back a wide shot's far face. + +Detection is OpenCV's YuNet, run on the full frame and on four overlapping +enlarged tiles, merged by non-maximum suppression, so a face only a few dozen +pixels wide is still seen. One track is kept and smoothed with an exponential +moving average; across a short detector miss the last box is held at a decaying +strength. The track restarts at each shot boundary the clip records, or, for a +clip that records none, where the picture jumps: a sharp drop in its HSV +colour histogram, or a spike in its difference from the previous frame (a cut +between two framings of one picture keeps its colours). A padded square +around the smoothed box is cut from every frame and resized to `crop_size`, then +the crops are padded to `modulus * n + remainder` frames (8n+1 by default) with mirrored warm-up and cool-down frames. + +Each frame carries a strength from 0 to 1: how much a face-detail pass should +change it. It is 1 when face width over frame width is at or below `gate_full`, +0 at or above `gate_zero`, linear between, and multiplied by the hold decay on +frames the detector missed. + +It returns `{crops, track}`. `crops` is a video (no audio) at the clip's frame +rate. `track` is a JSON record saved as its own `.json` file, readable by +`previous_result:.track`: + +| Field | Description | +| ----- | ----------- | +| `source` | `{width, height, frames, fps}` of the input | +| `crop_size`, `padding`, `gate_full`, `gate_zero`, `min_confidence` | The parameters used | +| `detector` | `{repo, file}` | +| `pad_before`, `pad_after` | Mirrored frames added to the start and end | +| `crop_frames` | Frames in `crops`, including the padding | +| `face_found` | Whether any frame had a face | +| `message` | Present only when no face was found | +| `frames` | One entry per source frame: `box` (the smoothed `[x, y, w, h]`, or null), `crop` (the `[x, y, side, side]` square cut), `strength`, and `state` (`tracked`, `held` or `none`) | +| `resets` | `[{frame, reason, detail}]` where the track restarted; `reason` is `shot`, `cut` or `lost`; a `cut`'s `detail` is `{histogram_correlation, frame_change}` | + +A clip with no face still succeeds: every strength is 0, the crops are the +centre square, and a `no_face_found` warning is raised. `validate_workflow` +refuses a `crop_size` that is not a multiple of 32, a `gate_zero` at or below +`gate_full`, a `padding` out of range, a `detector_repo` that is not a repo id +and a `detector_file` that is not a `.onnx` name, before anything downloads. + +### paste_face_track + +Put the crops of a face-detail pass back into the clip they were cut from, +using the `track` record `crop_face_track` returned: + +```json +{ + "name": "pasted", + "task": { + "command": "paste_face_track", + "arguments": { + "clip": "asset:interview.mp4", + "repaired": "previous_result:face_repair", + "track": "previous_result:face_crops.track" + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `clip` | Yes | The video `crop_face_track` tracked - frames, an audio+video pair, or the path or URL of a video file | +| `repaired` | Yes | The crops after a face-detail pass, still padded to the count `crop_face_track` produced; any frame size | +| `track` | Yes | The `track` record `crop_face_track` returned (`previous_result:.track`), or the path of its saved `.json`, such as an `output:` reference | +| `feather` | No | Fraction of the paste's radius that fades out, `0` to `1`. Default `0.3` | +| `color_match` | No | Shift each crop's mean colour onto the source's inside the mask before blending. Default `true` | + +The pad frames are dropped: the crop at `pad_before + i` goes back on source +frame `i`. Each crop is resized to the square it was cut from, then blended +over the frame with a radial feathered mask times that frame's strength. The +mask is the circle inscribed in the square; its outer `feather` fraction fades +from 1 to 0 along a smoothstep, so the square's corners and everything past +the circle keep the source. A square that ran past the frame's edge is +clipped to the frame. + +A frame with strength 0 (no face, or a face too large for the gate) is passed +through exactly. With `color_match`, the crop's mean colour is shifted onto the +source's inside the mask, so a repair pass that drifted in tone does not show +as a patch. + +It returns the source clip with its audio, frame rate and shots carried, the +same frames one for one. It refuses a track whose frame count or frame size +does not match the clip, a `repaired` with fewer frames than the track's +`crop_frames`, and a `track` that is not a `crop_face_track` record. +`validate_workflow` refuses a `feather` past `1` before the job runs. + +**Example:** [face-repair.json](../workflows/templates/ltx2/face-repair.json) — the repaired crops pasted back over the source, its soundtrack kept. + ### video_frames The frames of a generated video, as one `(frames, height, width, channels)` @@ -597,6 +1315,7 @@ a length still passes the whole track along: | `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | | `start_seconds` / `duration_seconds` | One pair | The slice in seconds; either may be omitted | | `start_frame` / `num_frames` / `fps` | One pair | The slice in video frames; `fps` is required, start and count may be omitted | +| `lead_frames` | No | Frame form only, default 0: extra audio before the cut. The slice starts at `start_frame - lead_frames` and still runs `num_frames`, so `start_frame` 48 with `lead_frames` 12 at 24 fps is audio from frame 36 for `num_frames` frames. A whole number, 0 or above; refused with the seconds form, and refused when `start_frame - lead_frames` is below zero (by `validate_workflow` for literals, at run time otherwise) | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | ### gain_audio @@ -1139,13 +1858,197 @@ scale as `rms_dbfs` (their powers sum to it), so the loudest band sits near `threshold_dbfs`. A silent track, or a band with no content at the track's sample rate, reads as `null` rather than `-inf`. +### analyze_beats + +Find where a song's beats fall, to cut picture to them - a Music 3 song +carries no tempo map, and a cut that lands on the beat needs one. An onset +envelope gives the tempo and the beats are tracked through it; the result is +JSON and nothing is built: + +```json +{ + "name": "beats", + "task": { + "command": "analyze_beats", + "arguments": { + "audio": "previous_result:song", + "anchors": [0.52, 31.9] + } + }, + "result": { "content_type": "application/json" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path, `asset:`/`output:` reference of an audio or video file, a waveform from a previous step, or an earlier step's generated audio or video | +| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | +| `tempo_bpm` | No | The tempo, when known. With exactly one anchor it lays an exact grid and nothing is detected; otherwise it steers detection toward that tempo | +| `anchors` | No | Where beats are known to fall, ascending: a list of times in seconds, or of `{beat_index, seconds}` marks placing detected beat `beat_index` (from 0) at `seconds`. One kind per list | +| `min_bpm` / `max_bpm` | No | The tempo range searched (default `60` / `200`); `min_bpm` must be below `max_bpm` | + +Returns `{bpm, beats, downbeat_phase, method, calibration, duration_seconds, +warnings}`: `beats` ascend, in seconds. `method` is `onset` (tracked from the +onsets), `rms_peaks` or `grid`. `downbeat_phase` is which of the first four +beats starts a bar, `null` when none stands out. `calibration` is +`{offset_s, drift, anchors_used}`, the anchors' shift at the first mark and +how much it changes by the last. `warnings` repeats what was emitted. + +Anchors correct the detection three ways: + +- Two or more warp the detected beats piecewise-linearly through them, so a + detection that starts late or drifts lands on the marks. +- One alone shifts every beat by the same amount. +- One plus `tempo_bpm` lays an exact grid through it, with no detection: + `method` is `grid`. + +A bare-seconds anchor snaps to the nearest detected beat, so it corrects a +drift of under half a beat; where the detection is further off than that, use +`{beat_index, seconds}` marks, which name the beat instead of guessing it. + +A track with no clear, regular onsets - a pad, a swell, a noise bed at any +level - falls back to the peaks of its loudness (`method` `rms_peaks`, with a +warning that no reliable beat was found and the peaks follow swells rather +than a pulse). A level that drifts (a fade, a loud first second) is not a +pulse. A pulse that tracks but is faint - its onsets repeat weakly at the +period and its beats sit low against the rest of the envelope, as in a song +with a long quiet build - is kept, with a warning to check a few beats against +the song and pass `anchors` or `tempo_bpm` if they are off. The tempo is the +track's strongest period, folded into the range: a half-time song whose pulse +is 50 BPM reports 100, not a riff that happens to repeat at 95. A silent track returns empty `beats` and a warning. So does a +near-silent one whose loudest 50 ms is under -40 dBFS - room tone, hiss, or a +song mixed far too low - since the log-compressed onset envelope would find a +pulse in it. Raise a real song's level first (`gain_audio`). + +`bpm` is always within `min_bpm`-`max_bpm`, or `null`. A pulse found outside the +range is folded by octaves into it (85.7 BPM searched at 140-200 reports 171.4, +beating every half pulse). Where no octave fits, the beats keep to a tempo in +the range and a warning names the pulse. Where the `rms_peaks` peaks have no +octave in the range, `bpm` is `null` with a warning. The beats run to the +song's ends: a hit at 0 s is a beat, and a noise bed starting at 0 s is not. + +Refused: `min_bpm` at or above `max_bpm`, and malformed anchors (mixed kinds, +out of order, a non-whole `beat_index`), at `validate_workflow`; an anchor past +the song's end, or a `beat_index` past the detected count, at run start. +It returns JSON, so `result` may only be `application/json`. + +### plan_cuts + +Plan a music video's cuts from a song's lyrics and beats. `transcribe_audio` +(with timestamps) says when each line is sung and `analyze_beats` says where +the beats fall; this turns the two into shots - each a start frame and a +length - that tile the song exactly, so one prompt per shot can be written and +the list rendered. The result is JSON and nothing is built: + +```json +{ + "name": "plan", + "task": { + "command": "plan_cuts", + "arguments": { + "transcript": "previous_result:transcribe", + "beats": "previous_result:beats", + "duration_s": "previous_result:beats.duration_seconds", + "lyrics": "variable:lyrics", + "segment_by": "line", + "fps": 24, + "min_scene_s": 1.5, + "max_scene_s": 10, + "vocal_tail_s": 0.5, + "snap_to_beats": true + } + }, + "result": { "content_type": "application/json" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `transcript` | Yes | `transcribe_audio`'s result with `timestamps` set (`"segment"` or `"word"`): a `{text, chunks}` dict of `{start, end, text}` chunks in seconds. A null `end` runs to the next chunk's start, or the song's end. Its `result` must be `application/json` | +| `lyrics` | No | The song's own lyrics, one line per line with a blank line or a lone section tag (`[Chorus]`) between stanzas, or a list of lines. The shots carry these lines verbatim and in order; the transcript only times them. `null` or `""` means none: the transcript's text is used | +| `beats` | For `segment_by: "beat"` | `analyze_beats`' result, or a list of beat times in seconds | +| `segment_by` | No | `line` (default), `stanza` or `beat` | +| `fps` | No | Frames per second the shots are counted in (default `24`) | +| `duration_s` | No | The song's length; `analyze_beats`' `duration_seconds` when `beats` is that whole dict, else the transcript's last end (with a warning). `"beats": "previous_result:beats"` passes only the result's `beats` list - the engine takes the key an argument is named for - so name the length too: `"duration_s": "previous_result:beats.duration_seconds"` | +| `min_scene_s` | No | The shortest a shot may be (default `1.0`) | +| `max_scene_s` | No | The longest a shot may be (default none) | +| `vocal_tail_s` | No | Seconds a sung shot holds past its last word (default `0`) | +| `include_instrumental_gaps` | No | Whether a long silence becomes its own instrumental shot (default `true`) | +| `min_gap_seconds` | No | The shortest silence that becomes its own shot (default `2.0`) | +| `snap_to_beats` | No | Whether every cut moves to its nearest beat (default `false`) | +| `modulus` / `remainder` | No | The model's render grid: a shot renders `modulus * n + remainder` frames (H3: `17`, `5`). With none, a shot renders exactly its cut | +| `min_frames` / `max_frames` | No | The fewest and most frames a shot may render (H3: `124`, `345`) | +| `lead_s` | No | Seconds a shot starts before its cut, as a run-up the model renders and `trim_video` drops. Each shot's `lead_frames` is `min(round(lead_s * fps), start_frame)`, so the first shot has none (default `0`) | + +Returns `{shots, bpm, fps, duration_s, total_frames, render_frames, +warnings}`. `shots` tile the song in order, each `{name, start_frame, +num_frames, cut_frames, lead_frames, lyric, kind}`: `start_frame` and +`cut_frames` are the cut's start and length on the timeline (the `cut_frames` +sum to `round(duration_s * fps)`), `lead_frames` the run-up before the cut, +and `num_frames` the length to render: `max(min_frames, lead_frames + +cut_frames)` raised to the grid, so the render starts at `start_frame - +lead_frames` and its cut is frames `[lead_frames, lead_frames + cut_frames)`. +With no grid arguments `num_frames` is `cut_frames` and `lead_frames` `0`. +`render_frames` is the sum of the `num_frames` - what the render costs, which +is more than `total_frames` when shots lead or round up. `lyric` the lines sung in it joined by newlines or `null`, and `kind` +`"vocal"` or `"instrumental"`. A dict rather than a list, since a list result +would become one artifact per shot. Its `result` may only be +`application/json`. + +How the plan is made: + +- With `lyrics`, those lines are the lines, and the transcript only lends them + timings: they are aligned to it word by word, since Whisper mishears sung + words and splits lines where it likes. A line never heard is placed between + its neighbours with a warning, as a shot of its own: when its neighbours + touch, it takes `min_scene_s` (at least half a second) from them, never + more than half of either. Blank lines and section tags are stanza + breaks, not sung lines. Without `lyrics`, each transcript chunk is a line. +- `segment_by` makes a shot per line, per stanza (the lyrics' blank-line + groups, or lines without a `min_gap_seconds` silence between them) or per + few beats (a cut on the first beat at least `min_scene_s` after the last). +- The silence between lines goes to the shot before it, or - when it lasts + `min_gap_seconds` and `include_instrumental_gaps` is on - becomes its own + instrumental shot, as can the silence before the first line and after the + last. +- A shot over `max_scene_s` splits evenly, each cut on its nearest beat when + there are beats, and a sung shot split this way is warned about by its + lyric, which every piece carries; one under `min_scene_s` merges into its + shorter neighbour. + A shot still outside the range is warned about by name. +- A shot whose render would pass `max_frames` is split, on a beat when + there is one, and warned about. +- `snap_to_beats` moves every cut to its nearest beat; with no beats it warns + and leaves the cuts where the lines put them. +- Every boundary is rounded once, from its absolute time, so the frame counts + sum to the song's and no rounding drifts. + +Refused, at `validate_workflow` where the value is a literal and at run start +otherwise: a bare-string `transcript` (the message names `timestamps` / +`return_timestamps`), `segment_by` not one of the three, `segment_by: "beat"` +without `beats`, `min_scene_s` above `max_scene_s`, and `fps`, `duration_s`, +`max_scene_s` at or below zero or `min_scene_s`, `vocal_tail_s`, +`min_gap_seconds` below zero, and a `remainder` outside `0` to below `modulus` (or one with no +`modulus`) or a `max_frames` below the smallest grid length at or above +`min_frames`. Also refused at run start: a song whose length +can't be known (no `duration_s`, no `beats` result, and no transcript to end +on) and `lyrics` with no sung lines. +[music-video-cuts.json](../workflows/templates/minimax/music-video-cuts.json) +chains `transcribe_audio`, `analyze_beats` and `plan_cuts`, passing H3's +grid (`17`, `5`, `124`, `345`) and a `lead_s` variable (default `0.5`, 12 +frames at 24 fps). Its shots are what +[music-video.json](../workflows/templates/minimax/music-video.json) renders: +the agent writes a prompt per shot, drops `lyric` and `kind`, and passes the +list as `shots`. + ## Assessment Probes Three read-only commands measure a finished cut and say where to look - `analyze_shots`, `analyze_seams`, `analyze_sync_drift`. Each is registered with `assessment=True` (`register_command`, `dw/tasks/task.py`), which is what makes a command a probe - not merely answering JSON, since -[`attribute_voices`](#voice-attribution) answers JSON too and is not one. +[`attribute_voices`](#voice-attribution) and [`check_script`](#script-check) +answer JSON too and are not probes. Each takes a video (a stored file - `asset:`, `output:` or a path, read straight from disk rather than decoded first - or the video an earlier step returned; not a @@ -1212,7 +2115,8 @@ Every seam between shots, audio and picture: | `seams[].between` | `[previous shot name, next shot name]` | | `seams[].seconds` | Where the seam sits in the file | | `seams[].kind` | `cut` or `dissolve` (a dissolve has `overlap_frames`) | -| `seams[].hard_cut` | Whether the incoming shot is marked `hard_cut: true` | +| `seams[].hard_cut` | Whether the incoming shot is marked `hard_cut: true` - `concat_videos` sets it on a shot opening a seam it butt-joined (a plain cut, or a cut with a `seam_fade_ms` fade). A seam that got an `audio_bleed_ms` bleed carries `audio_bleed_ms` instead of `hard_cut` | +| `seams[].audio_bleed_ms` | The bleed the seam actually got (clamped to the material); absent where none ran. The picture is still a cut, so `seam_frame_jump` stays quiet | | `seams[].before_shot_rms_dbfs` / `after_shot_rms_dbfs` | RMS level of the whole shot either side of the seam | | `seams[].level_step_db` | The absolute difference between those two shot levels. Shot against shot, not the audio at the seam's edges: a take's own tail and head can sit 20 dB apart, which is not a step the cut made | | `seams[].before_rms_dbfs` / `after_rms_dbfs` | RMS level of the 0.25 s either side of the seam - what `seam_hole`'s both-sides-voiced guard reads | @@ -1278,6 +2182,46 @@ without reading every command's schema. The list is exactly the commands declared `assessment=True`; `attribute_voices` stays in `commands` only, since it analyzes a song rather than checking a cut. +## Stem separation + +Split a mix into its htdemucs stems - `vocals`, `drums`, `bass` and `other` - +each returned as its own audio result. It is the separator `attribute_voices` +runs (one shared loader and cache), exposed for its own sake: transcribing the +`vocals` stem aligns sung lyrics better than transcribing the full mix, and +`drums` + `bass` + `other` is the instrumental to put under dialogue as a +score bed. + +```json +{ + "name": "stems", + "task": { + "command": "separate_stems", + "arguments": { "audio": "asset:song.mp3" } + }, + "result": { "content_type": "audio/wav", "file_name": "stems" } +}, +{ + "name": "lyrics", + "task": { + "command": "transcribe_audio", + "arguments": { "audio": "previous_result:stems.vocals" } + }, + "result": { "content_type": "application/json" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack | +| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | + +The result saves one file per stem, `-vocals`, `-drums`, `-bass` +and `-other`, and a later step reads one as `previous_result:.vocals`. +Stems are 44.1 kHz stereo, whatever the source was, and sum back to the mix. +The model is fixed (`htdemucs`, run in fp32), its weights download on first +use, and it needs the `demucs` package. On MPS a failed separation falls back +to the CPU and warns (`separation_cpu_fallback`). + ## Voice attribution Which reference voice sings each line of a song, by timbre - staging @@ -1319,7 +2263,7 @@ voiced, so a weak answer is visible as weak rather than silently accepted. | -------- | -------- | ----------- | | `audio` | Yes | The song - a path, `asset:`/`output:` reference, or an earlier step's audio or video | | `voices` | Yes | Each voice's name mapped to its reference: a list of `{start_seconds, duration_seconds}` (or `{start, end}`) spans into `audio`, or a path/`asset:` of a separate clip. At least 2 voices; names follow the variable-name pattern; each voice's reference must total at least `min_reference_seconds` | -| `lines` | No | Spans to attribute, `{start, end, text?}` (a `transcribe_audio` transcript, or its `chunks` list, drops in directly) or `{start_seconds, duration_seconds, text?}`. Omitted: the song is cut into fixed windows of `window_seconds` | +| `lines` | No | Spans to attribute, `{start, end, text?}` (a `transcribe_audio` transcript, or its `chunks` list, drops in directly) or `{start_seconds, duration_seconds, text?}`. A zero-length line (`start` = `end`, as Whisper's word timestamps sometimes give) comes back with `voice: null` rather than refusing the call. Omitted: the song is cut into fixed windows of `window_seconds` | | `windows` | No | Named spans to roll lines up into, `{name, start, end}`. Omitted: mirrors the fixed windows when `lines` is also omitted, otherwise none | | `window_seconds` | No | Length of the fixed windows used without `lines` (default `2.0`) | | `min_reference_seconds` | No | Least total reference length per voice; a shorter one is refused by name (default `3.0`) | @@ -1355,6 +2299,8 @@ well above that and well below the singing. | `min_voiced_seconds` | 0.5 s | A line or window under this much voiced time has no voice - `voice: null`, `uncertain: true` | | `uncertain_margin` | 0.05 | A line whose best score beats the runner-up by less than this is `uncertain` - the argmax is still reported | | `uncertain_share_margin` | 0.2 | A window whose leading voice's share beats the runner-up's by less than this is `uncertain` - a duet line, or a window straddling a hand-over | +| `piece_seconds` | 2.0 s | A line longer than this is also scored in pieces of this length | +| `mixed_line_share` | 0.25 | A line whose confidently attributed pieces give a second voice at least this share of their voiced time is `uncertain`, and its `reason` names each voice's seconds - one embedding of two singers can name either of them with a wide margin | | `voices_too_similar` | 0.8 | Two references scoring above this against each other make every line between them a weak answer whatever its scores say | A `voices_too_similar` pair is reported in `reference_similarity`, added to @@ -1369,6 +2315,125 @@ and are cached between calls like any other model. On MPS a `separate: true` run that fails in htdemucs falls back to the CPU and warns (`separation_cpu_fallback`) rather than failing the step. +**Example:** [attribute-lines.json](../workflows/templates/attribute-lines.json) — Which singer sings each line of a song, and when: `transcribe_audio` with timestamps, its transcript handed whole to `attribute_voices` as `lines`. + +### Checking the lip-sync target + +A sung multi-shot piece can put the right song on the wrong mouth: the shot +lip-syncs whoever the prompt or the reference made most prominent, not the +singer the song gives that line to. Listening does not catch it, and one +frame per shot does not either. This check lines each sung line up with the +picture and looks at whose mouth is open on it. + +1. Run `attribute-lines` on the song with each singer's reference spans as + `voices`. Every line comes back with numeric `start` and `end` seconds into + the song, its `voice`, `margin` and `uncertain` flag. +2. Map each line's song times onto the cut's timeline (the rules below). +3. For every line with a non-null `voice`, take two moments: `start + 0.3 s`, + where the mouth has opened on the line's first syllable, and the line's + midpoint. +4. Make one `get_output_frames` call per face: `at` the moments, `crop` the + face's region, `hear: 1.0`. At most 32 moments per call, so a long song + takes more than one call per face. +5. On each line, the singer `voice` names should be the face with an open + mouth. Another face singing it is the wrong lip-sync target; both mouths + open on a solo line is a shot that lip-syncs everyone. +6. Skip the lines that are `uncertain` or have a null `voice`, and name them + in the report - a skipped line is unchecked, not passed. + +**Song time to cut time.** `attribute-lines` answers in seconds into the song +it was given; the frames are read from the cut. The offset between the two +depends on how the piece was assembled: + +- `minimax/music-video`: song time is cut time unless `trim_frames > 0`. The + template slices the song `lead_frames` before each shot's `start_frame`, + renders `num_frames`, keeps the shot's `cut_frames` with `trim_video`, joins + the trims with `concat_videos` at `trim_frames: 0`, and lays the whole song back with + `pair_audio` at `fit: video`, so no offset applies. A trim drops frames at + every seam, and the song no longer lines up past the first one. +- `join_into_song`: add `start_frame / fps − cue_seconds` to every line time, + where `start_frame` is the first sung shot's, from the step's shots output. + That shot's frame 0 is where song time `cue_seconds` lands. +- Any other assembly: transcribe the final cut's own track instead, so the + times are already the cut's. Attribution is weaker there - dialogue ducked + under the song is in the mix too. + +**Where it misleads:** + +- A duet line - two singers on one line, or call and response inside one + Whisper segment - comes back `uncertain`: its 2 s pieces name more than one + voice, and the `reason` says how many seconds each holds. + `timestamps: "word"` splits it into words, each attributed on its own, at + the cost of noisier scores. A zero-length word comes back with a null + `voice`, like any line too short to embed. +- Whisper can return one segment over most of a song, its text a repeated + hallucination, when the accompaniment drowns the words. Such a line is a + duet line by the rule above whenever two singers are in it; re-run with + `timestamps: "word"`, or transcribe the `vocals` stem of + [`separate_stems`](#stem-separation) instead. +- `start + 0.3 s` is a heuristic: a singer who opens late or holds a pickup + note is still closed-mouthed there. That is why the midpoint frame is read + too; trust the two together, not either alone. + +### ingredients_grid + +Lay individual images - a character, a prop, a location - out on one canvas as a +reference sheet. It is `frame_grid`'s counterpart for separate images, and builds +the still `ltx2/reference-sheet` conditions on. Pure PIL. + +```json +{ + "name": "sheet", + "task": { + "command": "ingredients_grid", + "arguments": { + "images": ["asset:hero.png", "asset:sword.png", "asset:castle.png"], + "width": 768, + "height": 448 + } + }, + "result": { "content_type": "image/png" } +} +``` + +| Argument | Default | Description | +| -------- | ------- | ----------- | +| `images` | required | Images in reading order: references, paths or `{"location": ...}` dicts; `gather:` of a `for_each` step splices in | +| `width`, `height` | 768, 448 | Canvas size in pixels | +| `layout` | `auto` | `rows`, `panels` or `auto` | +| `fit` | `contain` | `contain` scales each image to fit whole and pads with `background`; `cover` fills its cell and crops the overflow | +| `gap` | 8 | Pixels between cells, and kept clear of the canvas edge | +| `background` | `white` | Canvas colour, a name or `#hex` | +| `max_images` | 12 | More images than this is refused, never silently dropped | + +`rows` keeps every image at its own aspect ratio and justifies each row to the +canvas width. Which images share a row is searched: every way of cutting the +list, in order, into rows is scored on wasted canvas, overflow past the height, +and spread of row heights. `panels` gives every image an equal grid cell, the +column count chosen to waste the least canvas, with a short last row centred. +`auto` uses `rows` for up to 3 images and `panels` for more. Transparency is +flattened onto white. Returns one RGB image of exactly `width` x `height`. + +To build the sheet inside `ltx2/reference-sheet`, add this step first and point +`static_sheet`'s `video` at it (`"video": "previous_result:sheet"` in place of the +`reference_sheet` asset). Keep `width` and `height` the clip's own, so the sheet +reads at the size the model was trained on: + +```json +{ + "name": "sheet", + "task": { + "command": "ingredients_grid", + "arguments": { + "images": ["asset:hero.png", "asset:sword.png", "asset:castle.png"], + "width": "variable:width", + "height": "variable:height" + } + }, + "result": { "content_type": "image/png", "save": false } +} +``` + ## Data Gathering ### gather_images @@ -1429,8 +2494,8 @@ can be upscaled without losing what was generated alongside it: Captioning (`image_to_text`) is the exception - describe a frame, taken with `get_first_frame`, rather than a video. -A file path or `asset:`/`output:` reference to a video is not accepted here, -even though the route above takes a video from a step - route it through +A file path or `asset:`/`output:` reference to a video is not accepted here +(`grade`, whose input is `media`, is the exception), even though the route above takes a video from a step - route it through `video_frames` first: an image command's `image` argument must be an `AudioVideo`, a frame array, or a still image. @@ -1830,7 +2895,7 @@ The result needs no `sample_rate`. A generated track carries the rate its model ### Generating a voice to condition on -The role this earns its place in is voice *timbre reference*, not the track a mouth follows. MiniMax H3 lip-syncs well when it generates the speech itself and poorly when it must follow supplied audio, so its `MiniMaxH3AudioReference` takes a few seconds of a voice to fix timbre, pitch and delivery while H3 still generates the line. Build the reference with `from_previous_result` and the clip's own sample rate comes across with it: +The role this earns its place in is voice *timbre reference*, not the track a mouth follows. A `MiniMaxH3AudioReference` is audio H3 reads, not audio it plays: it takes a few seconds of a voice to fix timbre, pitch and delivery while H3 still generates the line, and a mouth asked to follow a whole track through a reference follows it loosely. That reference is still the better lip sync: `music-video` and the `match_audio` chain templates pass each shot's slice as one. Holding the track with `hold_audio` (the `workflows` guide, "H3: generating to a held soundtrack") keeps it exact, but on a sung track the mouth followed it less well than the reference (#619's A/B, measured in that section), so hold is opt-in. Build the reference with `from_previous_result` and the clip's own sample rate comes across with it: ```json "references": [ @@ -1875,10 +2940,80 @@ Transcribe spoken audio to text with a local Whisper-class model. The word-corre Multi-channel audio is downmixed to mono and resampled to 16 kHz before transcription, since that is what a Whisper-class model is trained on; the source audio itself is untouched. By default the result is plain text, read with MCP's `get_output_text`. -Set `timestamps` to `"segment"` or `"word"` to get chunk timings instead — a music video cut to the lyric, or a dialogue shot checked against its line, needs the times Whisper already produces past 30 s rather than the collapsed string. The result becomes `{"text": ..., "chunks": [{"start": ..., "end": ..., "text": ...}, ...]}`, so the step's `result.content_type` must be `"application/json"` rather than `"text/plain"`, and it's read with MCP's `get_output_text` (JSON results are text). A clip under 30 s asks Whisper for timestamps explicitly when `timestamps` is set — the 30 s long-form threshold is a separate, unrelated reason to ask. +Set `timestamps` to `"segment"` or `"word"` to get chunk timings instead — a music video cut to the lyric, or a dialogue shot checked against its line, needs the times Whisper already produces past 30 s rather than the collapsed string. The result becomes `{"text": ..., "chunks": [{"start": ..., "end": ..., "text": ...}, ...]}`, so the step's `result.content_type` must be `"application/json"` rather than `"text/plain"`, and it's read with MCP's `get_output_text` (JSON results are text). A clip under 30 s asks Whisper for timestamps explicitly when `timestamps` is set — the 30 s long-form threshold is a separate, unrelated reason to ask. Every `start` and `end` is a number of seconds: a chunk the clip ends inside - a song cut mid-line, which Whisper leaves open-ended - ends at the clip's duration, so the transcript drops straight into [`attribute_voices`](#voice-attribution)' `lines`. `"word"` can give a zero-length chunk (`start` = `end`) on a short or clipped word; `attribute_voices` reads it as a line with no voice. + +To compare a take against the lines it was meant to speak, use [`check_script`](#script-check) rather than reading the transcript by eye. **Example:** [transcribe-audio.json](../workflows/templates/transcribe-audio.json) — Transcribe an audio file to text. +## Script check + +Whether a dialogue take speaks its script. Confirming it used to mean transcribing the take and comparing the transcript to the script by eye, which missed a word Whisper dropped, an H3 tag read aloud, and a last word cut off by the end of the file. `check_script` does the comparison and reports where to look. + +It transcribes the take with [`transcribe_audio`](#speech-transcription) word timestamps - the same code and model cache, no second ASR path. Every heard word is measured, and one at or below the dead-air floor is discarded as unheard, because Whisper invents words over silence. So is every word of a repetition loop - the same words over and over (`Pre-pre-pre-...`), which Whisper invents over music or room tone loud enough to pass the floor. H3 markup is stripped from each expected line, and the heard words are aligned to the expected words in order (`difflib`). Each line scores the similarity of its own expected and heard words. It decides nothing: findings are places to look, not verdicts. It is not an assessment probe (`assessment=True`); like [`attribute_voices`](#voice-attribution) it answers a JSON document. + +```json +{ + "task": { + "command": "check_script", + "arguments": { + "audio": "output:take.wav", + "lines": ["[English] Where were you?", "I was at the station."] + } + }, + "result": { "content_type": "application/json" } +} +``` + +The step's `result.content_type` must be `"application/json"`; read the answer with MCP's `get_output_text`. + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | The take - a path, `asset:`/`output:` reference, or an earlier step's audio or video (its soundtrack is taken) | +| `lines` | Yes | The expected lines in order: a list of strings or `{text, shot}` objects, `shot` naming the take's shot the line belongs in. Markup is stripped (below). `[]` means no speech is expected | +| `shots` | No | The take's shot map, `[{name, start_frame, num_frames, start_sample, num_samples}]`. Default: the take's own `.shots` (an earlier step's video), else the run manifest or `keep_output` sidecar beside an `asset:`/`output:` file - resolved as the assessment probes resolve them (`assess.resolve_shots`). The argument wins over both | +| `similarity` | No | Least similarity, 0..1, of a line's heard words to its expected words before it is a `line_mismatch` (default `0.85`) | +| `model_name` | No | HuggingFace ID of a Whisper-class ASR model (default `openai/whisper-base`) | +| `sample_rate` | No | Sample rate of a waveform passed directly | +| `device` | No | Where the ASR model runs | + +Markup stripped from an expected line: `[Language] ...`, ``, ``, `[unclear]`, speaker IDs `(S1)` and `(S1,S2)`, and any other `` or `[tag]`. A plain parenthetical stays dialogue. The words of the stripped tokens (`cutoff`, `unclear`, `english`, `s1`) are what `tag_spoken` listens for; a tag word heard is reported there and left out of the line's similarity, so it is never a `line_mismatch` too. + +| Rule | Fires when | `value` / `threshold` | `at` | +| ---- | ---------- | --------------------- | ---- | +| `line_mismatch` | A line's similarity is below `similarity`; a dropped line comes back with `heard: ""` | The line's similarity / `similarity` | `{line, seconds, word}` - the line's first heard word, or where it should have been | +| `tag_spoken` | A word from the line's stripped markup was heard, and is not also a word of the line's dialogue | The word heard / none | `{line, seconds, word}` | +| `line_clipped_at_end` | A line's last heard word ends inside the final `CLIP_TAIL_SECONDS` of its shot (when the line has one) or of the file, and that tail is still above `GUARD_FLOOR_DBFS` | The tail's level in dBFS / `GUARD_FLOOR_DBFS` | `{line, seconds, word, shot}` - `shot` when the line has one | +| `speech_where_silent` | `lines` is `[]` and words were heard above the floor and outside a repetition loop | Number of words heard / 0 | `{line: null, seconds, word}` - the first; `shot` when shots are known | +| `speech_in_silent_shot` | Shots are known, at least one line names a shot, and words were heard above the floor and outside a repetition loop in a shot no line names. A word is in the shot its midpoint falls in; one a line in another shot matched (a word straddling the cut) is that line's, not counted | Number of words heard in the shot / 0 | `{line: null, seconds, word, shot}` - the shot's first counted word, `seconds` no earlier than the shot's start | + +Every finding is severity `warn`; `at.shot` is set whenever the shot is known. A rule that does not apply to the call is listed in `rules_skipped` with its reason: `speech_in_silent_shot` is skipped when no shots are known (`line_clipped_at_end` then looks at the file's end only), when the shots carry neither a sample span nor a frame span the take's frame rate can place, or when no line names a shot (no shot is known to be meant silent). + +The thresholds are module constants in `dw/tasks/script_check.py`, deliberately not in `RULES` (`dw/assessment_rules.py`): every rule there must fire on a synthetic file that holds no script. The answer echoes them under `thresholds`. + +| Constant | Value | What crossing it does | +| -------- | ----- | --------------------- | +| `DEFAULT_SIMILARITY` | 0.85 | A line below this similarity is a `line_mismatch` (the `similarity` argument overrides it) | +| `GUARD_FLOOR_DBFS` | -65 dBFS (= `DEAD_AIR_FLOOR_DBFS`, `shot_dead_air`'s floor) | A heard word whose loudest window is at or below it is discarded as unheard; also the level a clipped tail must exceed | +| `GUARD_WINDOW_SECONDS` | 0.05 s (= `DEAD_AIR_WINDOW`, `shot_dead_air`'s window) | The window a heard word is measured in - its loudest one, so a span overhanging a pause does not average a real word away | +| `CLIP_TAIL_SECONDS` | 0.25 s | The final stretch of the file a line's last word must end in to be `line_clipped_at_end` | +| `REPEAT_RUN_MIN` | 6 | Heard words repeating the same unit this many times over, back to back, are a repetition loop: every chunk holding one is discarded (`reason: "repetition"`), however loud | +| `REPEAT_MAX_PERIOD` | 4 words | The longest repeating unit a loop is looked for in - `pre pre pre` has period 1, `thank you thank you` period 2 | + +The result: `findings`, `lines[]` (`expected`, `heard`, `similarity`, `start`, `end`, `shot`), `discarded[]` (`text`, `start`, `end`, `level_dbfs`, `reason` - `below_floor` or `repetition`), `unmatched[]` (heard words aligned to no line), `transcript`, `model_name`, `rules_applied`, `rules_skipped[]` (`rule`, `reason`) and `thresholds`. A line's `shot` is the shot it names, else the shot its heard words overlap most (null when none is known). `shots` is the map as placed - `[{name, start, end}]` in seconds on the take, or null when none is known - and `shots_source` says where it came from: `argument`, `artifact`, `manifest` or `none`. + +How the alignment reads a take: + +- A dropped line comes back with `heard: ""` and a `line_mismatch` at where it should have been. +- An added line's words go to `unmatched` and lower no line's similarity. +- A line spoken out of order reads as a mismatch on that line. +- Whisper mishears names and numbers (`2` for `two`), so listen before re-rolling a take on a mismatch. +- A quiet line can be discarded by the guard. Discarded words are reported under `discarded`, not hidden. + +A malformed `lines` - not a list, an entry without text, an unknown key in a line object - is refused at validation, at `steps[i].task.arguments.lines`, as is a `similarity` outside 0..1. A line naming a shot the map lacks is refused, listing the map's shots, and so is a line naming a shot the map holds twice; a literal `shots` and literal lines are checked against each other at validation, and a `shots` or `lines` that is a reference is checked when the step runs. + +**Example:** [check-script.json](../workflows/templates/check-script.json) — Check a take against its script and save the answer as JSON. Its defaults are `openai/whisper-large-v3-turbo` at `similarity` 0.6, not the task's `openai/whisper-base` at 0.85: on the two-model measurement (#609, job `af5de241ca83`), turbo scored the take's correct lines 0.71 to 1.00 and its wrong or missing lines 0.00, where base scored correct lines as low as 0.43 on mishearings (`O'Connor`, `415`). Turbo writes numbers as words (`four fifteen`), so a script with digits costs similarity under it. + ## Frame Interpolation Increase video frame rate using RIFE (Real-Time Intermediate Flow Estimation). Takes a video and inserts intermediate frames between each pair. The result is one video artifact without a soundtrack - the frame count changed, so [`pair_audio`](#pair_audio) is how the original track comes back. [interpolate-frames.json](../workflows/templates/interpolate-frames.json) shows the interpolation itself. @@ -2093,3 +3228,4 @@ Canny edge detection followed by ControlNet generation: - [generate-speech.json](../workflows/templates/generate-speech.json) — Speak a line with a local text-to-speech model - [voice-timbre-reference.json](../workflows/templates/minimax/voice-timbre-reference.json) — Generate a voice and condition H3's `