Skip to content

feat(canvas): noise adjustment for film grain - #9933

Merged
joshistoast merged 11 commits into
mainfrom
feat/canvas-noise-adjustment
Oct 23, 2026
Merged

joshistoast merged 11 commits into
mainfrom
feat/canvas-noise-adjustment

Conversation

@joshistoast

@joshistoast joshistoast commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds a Noise adjustment for raster layers and groups, aimed at film grain.

  • Distribution: Gaussian or Uniform, with the same standard deviation at the same amount.
  • Grain: Mono (one value shared by R, G, B), Color (independent per channel) or Chroma (OKLab a/b only, lightness kept).
  • Tonal response: Midtones (default) scales grain by 4·Y′(1 − Y′), so it is full at mid-grey and fades to none at pure black and white, as film grain does. Even keeps the same strength at every tone.
  • Size: 1–8 px, about the grain's diameter. Larger sizes blur the sample field with a Gaussian of σ = √(size² − 1)/2, renormalized so strength does not change with size. Size 1 is per-pixel grain.
  • Amount: 0–100%, default 10%. σ = amount × 128 on the 0–255 scale (amount × 0.32 on OKLab a/b for Chroma).
  • New pattern: the dice button re-rolls the seed and is undoable. Duplicating a noise adjustment gets a fresh seed; duplicating a layer keeps it.

Noise is a deterministic hash of seed, pixel position and channel. A layer's grain lives in layer-local pixels and moves, scales and rotates with the layer; a group's grain is anchored to document pixels. Adjusting any sub-rectangle at its own origin gives exactly the pixels a whole-surface pass gives there, at every size, so partial redraws, exports, bakes and PSD all match the canvas.

A layer's own stack runs on layer-local pixels before resampling on the display, in merge/crop/copy bakes and in PSD. Generation and composite export do the same whenever the stack contains noise, and otherwise unless the layer is downscaled: a color-only stack on a downscaled layer, with no current display copy to reuse, is adjusted after resampling so the work stays bounded by the output size. Noise never takes that shortcut, because resampling must average its grain exactly as on screen. Merge Down bakes both layers' stacks into the merged pixels, which also fixes color adjustments being dropped on merge. Rasterize bakes a stack that contains noise into the source render before the transform and removes it, so the grain stays where it was.

Related Issues / Discussions

Tonal response, grain size and chroma grain follow ideas from the FilmGrain nodes in JPPhoto's cinematic-effects-pack.

QA Instructions

  • pnpm check:release in invokeai/frontend/webv2: lint, unit (10,904), browser, fixtures, architecture and browser performance, project-file journeys and accessibility all pass. Across three full runs, the browser suite failed one different, unrelated test each time (hotkey modal, workflow style-preset picker, gallery starred strip) under machine load; each passes 3/3 alone and none is touched by this branch.
  • Manual check against the mock backend (pnpm dev:mock):
    1. Draw a black-to-white gradient, right-click the layer, choose Add adjustment → Noise. The row reads "Noise 10% · Mono", and grain fades out toward pure black and white.
    2. In Properties, switch Grain to Chroma (color speckle, lightness unchanged), Tonal response to Even (grain reaches the extremes), and raise Size with Shift+→ (1 px steps). Each edit updates the canvas and undoes cleanly.
    3. Merge Down a noisy layer onto another, and Rasterize a transformed noisy shape or text layer: the canvas looks the same before and after, and undo restores the adjustments.
    4. Repeat on a layer group.
  • Noise pass cost, Node 22, median of 7, 4096² ImageData:
Stack ms
LUT only 121
Noise size 1, Even, mono / color 76 / 89
Noise size 1, Midtones, mono / color 114 / 123
Chroma size 1, Even / Midtones 499 / 522
Noise size 2 / 4 / 8, mono 305 / 436 / 716
Noise size 8, Midtones, color / chroma 1909 / 1685
  • Generation composite of a 4096² noisy layer into a 1024² output, Chromium, median: 8 ms at scale 1, 27 ms at scale 0.5, 116 ms at scale 0.25. Cost follows the sampled layer region, not the output.
  • Performance baselines: architecture-baseline.json adds render/thumbnail.ts to the editor's source-owner set. The bundler moved that small, side-effect-free module from a lazy chunk into an existing eager one after the kernel grew; no import changed and request counts are unchanged. browser-baseline.json raises only the byte figures that failed on the tree merged with main, to their measured values, since main's growth and the larger kernel together exceed the allowance: editor-canvas activated largest asset 305,523 → 311,154, editor-canvas script 4,718,489 → 4,767,087 and total 5,014,895 → 5,065,152, and editor-workflow script 4,644,510 → 4,691,655 (the kernel sits in a chunk every editor route loads). Timings are unchanged.

Review

Resolved:

  • Merge Down composited raw cache pixels and lost both layers' adjustments. It now bakes each layer's stack layer-local before compositing, reusing the display's current adjusted copy when it has one.
  • Older builds could drop a noise document's whole adjustment stack and autosave the loss. Noise documents are now canvas schema 5, which older builds refuse (see Compatibility).
  • A legacy project kept whole because its workflow repeats an item id saved without raising the canvas floor, so Noise could land in schema 5 under an older floor. Whole saves now raise it like split saves.
  • Rasterize ran noise on the new pixel grid after the transform bake, changing the pattern. Noise stacks are now baked before the transform.
  • Wide text under heavy downscaling could allocate past its composite memory reservation. Rendering now reserves the excess over the estimate before allocating, and refuses cleanly as over-budget when it does not fit.
  • Generation and export share one over-budget error, with the user-facing message unchanged.

Remaining limits:

  • Generation and composite export adjust a downscaled layer's color-only stack after resampling when they cannot reuse the display copy, keeping memory bounded by the output. A nonlinear adjustment (curves, levels, gamma) on such a layer can therefore differ slightly from the canvas preview. Adjusting layer-local first would remove the difference at a layer-sized memory cost; this predates the PR.
  • Scrubbing Size or Chroma on a very large layer rebuilds its adjusted copy on the main thread once per changed preview frame (see the table above). Moving the full-rebuild path to a worker pool is planned as a follow-up.
  • Rasterizing text still sizes its output from estimated bounds, so text wider than its estimate is clipped. That predates this PR and is not noise-specific.
  • Generating with a very large, heavily downscaled noisy layer can return over-budget. Generation reads a detached snapshot, so it cannot reuse the display copy.
  • A merge, crop or copy of a large downscaled adjusted layer with no current display copy reserves a layer-sized copy and can return over-budget where it previously succeeded.
  • PSD cannot represent group adjustments, so a group's noise is baked onto each member separately. It can differ where semi-transparent members overlap.
  • A dragged floating selection keeps the grain it had when lifted until it is placed.
  • Thumbnails without a current display copy approximate the grain by scaling amount and size.

Compatibility / Rollout

  • Adds a noise arm to the persisted adjustment schema. Existing documents load unchanged.
  • A document with any noise adjustment, enabled or not, on any layer or group, is written as canvas schema 5, and saving it raises the project's canvas floor to 5, whether the project is stored split or kept whole. The version is sticky: removing the noise does not lower it. Builds whose maximum is 4 refuse the project, and its .invk export, as made by a newer version, instead of loading it without the stack. The backend needed no change; its existing floor check refuses clients below the floor.
  • Documents without noise keep writing schema 3 or 4 as before.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

Adds a Noise raster/group adjustment with Gaussian or Uniform distribution, monochromatic or color grain, amount, and a re-rollable seed.
Noise is a deterministic function of position, so display, generation, export, merges and PSD share the same grain.
Layer bakes and PSD now apply a layer's own adjustments before resampling, matching the display.
@github-actions github-actions Bot added frontend PRs that change frontend files docs PRs that change docs labels Oct 19, 2026

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers

  1. [P1 · Must fix] Merge Down drops Noise. mergeLayerController.ts:139 composites raw cache pixels; the reducer removes the layer adjustments. Impact / recovery: Grain disappears from either adjusted layer; undo is the recovery. Test: Add a Noise-specific pre/post Merge Down pixel comparison. Fix: Bake adjustments before compositing. If skipped: Merge Down violates the PR’s promise that merges preserve grain.

  2. [P1 · Must fix] Older builds can erase the whole adjustment stack. canvasMigration.ts:125 accepts Noise, while the writer still emits schema 3 or 4. An older client can drop the unknown stack on load, then autosave that loss. Impact / recovery: Permanent after the save. Test: Open and save a Noise document with an older schema reader. Fix: Mark Noise documents with a schema version older builds refuse. The PR documents this limitation, but it remains a data-loss path.

Other findings/issues

  1. [P2 · Should fix] Rasterize changes the Noise pattern. rasterizeLayerController.ts:86 transforms raw parametric pixels, resets the transform, then keeps the adjustment. Noise runs on a new pixel grid. Impact / recovery: Grain changes after rasterizing a transformed text, shape or gradient layer; undo recovers it. Test: Compare pixels before and after rasterizing a transformed noisy layer. Fix: Apply Noise before baking the transform.

  2. [P2 · Should fix] Text noise can exceed its memory reservation. rasterComposite.ts:247 reserves from estimated text bounds, while rasterization uses browser-measured bounds. Wide text under heavy downscaling can allocate a larger scratch region than reserved. Impact / likelihood: Possible budget overrun or browser allocation failure with wide text and tight memory. Test: Use text whose measured bounds exceed the estimate. Fix: Reserve from actual layer bounds or a conservative text bound.

Noise gains a 1–8 px grain size (coordinate-pure Gaussian-correlated samples), a midtones response that fades
grain to none at black and white, and a chroma grain on OKLab a/b that keeps lightness.
Re-records the editor source-owner set and canvas route bytes for the larger kernel.
Any noise adjustment, enabled or not, raises the canvas and the project floor to 5 and never lowers it,
so older builds refuse the project instead of dropping the adjustment stack and saving the loss.
Merge Down bakes both layers' stacks layer-local before compositing; Rasterize bakes a noise stack before the
transform and drops it. Composites reserve measured text scratch past its estimate or refuse as over budget.
Covers grain size, tonal response and chroma, how Merge Down and Rasterize bake adjustments, the noise kernel's
cost and region-independence invariants, and that older versions refuse projects with noise.
…ustment

# Conflicts:
#	invokeai/frontend/webv2/performance/browser-baseline.json
#	invokeai/frontend/webv2/src/workbench/projects/projectFile.test.ts
@joshistoast
joshistoast requested a review from JPPhoto October 23, 2026 01:14

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge Down, Rasterize, and text-reservation fixes hold. But...

Merge blockers:

  • P2: invokeai/frontend/webv2/src/workbench/projects/durableSyncedPersistence.ts:1465-1473: A legacy whole project with repeated workflow IDs can save Noise as canvas schema 5 without raising minimum_canvas_schema_version. The backend preserves the old floor, so older builds see the project as compatible until they try to open it.
    Effect: The saved compatibility floor breaks the PR's promise to raise it to 5.
    Likelihood: Every affected legacy whole project; uncommon overall.
    Recovery: Save again after the floor is raised.
    Test: Save Noise in the repeated-ID whole-project fixture; assert the request and stored floor are 5, and an older client is refused.

Other findings/issues:

  • P2: invokeai/frontend/webv2/src/workbench/canvas-engine/render/rasterComposite.ts:351-365: When a layer is downscaled, nonlinear color adjustments can run after resampling. Generation supplies no adjusted surface (invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.ts:183-193), so its pixels can differ from the display, despite the PR's promise that adjustments run before resampling on every path.
    Effect: The generated base image can differ from the canvas preview.
    Likelihood: Any generated composite with a downscaled layer using a nonlinear color adjustment.
    Recovery: Apply these adjustments to layer-local pixels before resampling, or narrow the stated promise.
    Test: Compare display and generation pixels for a downscaled raster with a gamma or levels adjustment.

A legacy project kept whole because its workflow repeats an item id now sends the raised canvas floor like split
saves do, so saving Noise into it stores floor 5 and older clients refuse it.
@joshistoast
joshistoast requested a review from JPPhoto October 23, 2026 01:47

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers:

  • P2: invokeai/frontend/webv2/src/workbench/canvas-engine/engine.ts:2552-2553: On an export cache miss, getAdjustedCacheSurface() builds a full layer-sized adjusted surface. exportRasterComposite reserves only the output and sampled scratch (invokeai/frontend/webv2/src/workbench/canvas-engine/exportRasterComposite.ts:57-59). Exporting a small region of a large downscaled noisy layer can therefore allocate beyond its reservation; the budget controller records the cache but cannot refuse or extend that lease.
    Effect: Export may exceed the raster budget or fail allocation instead of returning over-budget.
    Likelihood: Medium for large layers, small export regions, and tight budgets.
    Recovery: Retry after cache eviction; document state is unchanged.
    Test: Export a small region of a large downscaled noisy layer with no adjusted cache; verify it uses only the reserved sampled scratch path.

Export reuses the display's adjusted copy only when it is current and never builds a layer-sized one, so a cache miss adjusts just the reserved sampled region.
@JPPhoto
JPPhoto self-requested a review October 23, 2026 12:43

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved!

Main's growth and the Noise adjustment together exceed the 1% byte allowance on the editor build and on canvas activation.
@joshistoast
joshistoast merged commit 2af04c9 into main Oct 23, 2026
21 checks passed
@joshistoast
joshistoast deleted the feat/canvas-noise-adjustment branch October 23, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs PRs that change docs frontend PRs that change frontend files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants