Bake flat-field profiles to a self-contained file store

CLAUDE.md

@@ -81,3 +81,4 @@ Every feature lives in `negpy/features/<name>/`:
 
 - **CPU/GPU parity**: any change to a stage's math must land in both `logic.py` and its `.wgsl` shader. Constants mirrored as WGSL literals (histogram bins, zone density, metrics offsets) have parity tests — keep them in sync.
 - **Working-space OETF + luminance row are inlined** in `lab_sharpen_h.wgsl` and `rl_init.wgsl` (Adobe RGB 1998 gamma 563/256, D65 Y row) — a TRC or primaries change must update them, not just `kernel/image/logic.py`. `LabUniforms` is declared in 6 lab shaders (lab, lab_sharpen_h/v, rl_blur_h, rl_div_v, rl_mult_v) — any field change touches all six plus the `struct.pack` in `gpu_engine.py`, and the trailing `_pad*` floats keep the block at 48 bytes. `rl_init.wgsl` binds no uniform at all (the auto layout prunes it).
+- **Flat-field gains resolve through a provider, not the config.** The per-image `FlatFieldConfig` carries only an opaque `profile_id`; the baked gain map lives in a per-profile `.npz` in `APP_CONFIG.flatfield_dir` (`services/assets/flatfield.py`, the sensor/crosstalk file-store pattern). `apply_flatfield`/`flatfield_token` look the gain up via `set_gain_provider`, which `desktop/main.py` wires at startup — any render path outside the desktop app (a script, a headless test exercising flat-field) must call `set_gain_provider` first or the correction silently no-ops. Legacy DB profiles (the retired `flatfield_profiles` table) are one-shot migrated by `flatfield_migration.py`.

docs/PIPELINE.md

@@ -21,7 +21,7 @@ Here is what actually happens to your image. We apply these steps in order, pass
 
 *   **Physical Model**: We treat the input as a **radiometric measurement**. Pixel values represent linear transmittance captured by the sensor.
 *   **Source corrections** (linear domain, before the log conversion):
-    *   **Flat-field** (`negpy.features.flatfield`): divides out illumination falloff using a blank reference frame. A per-channel gain map $\text{mean}(\text{blur})/\text{blur}$ (computed on a 256 px copy, clamped to $[0.25, 4]$) multiplies the linear source. The reference's identity is folded into the render's source hash.
+    *   **Flat-field** (`negpy.features.flatfield`): divides out illumination falloff using a blank reference frame. A per-channel gain map $\text{mean}(\text{blur})/\text{blur}$ (computed on a 256 px copy, clamped to $[0.25, 4]$) multiplies the linear source. The reference is decoded and the gain **baked once** into a profile (an `.npz` in `APP_CONFIG.flatfield_dir`, keyed by an opaque id), so the render never touches the original reference file — moving or deleting it is harmless. The per-image edit stores only the profile id; the render path resolves the gain through a provider (`set_gain_provider`, wired to `services/assets/flatfield.py` at startup) and caches it, and the profile id + a content token of the gain are folded into the render's source hash.
     *   **Sensor crosstalk unmix** (`sensor_matrix`, `features/process/sensor.py`): for single-shot narrowband camera scans, the camera's CFA passbands overlap the light source's bands, so a pure R/G/B exposure leaks into the other channels. That is a fixed property of the sensor+light pair, independent of film. It is calibrated once from three bare-light exposures (response columns normalized to a unit diagonal so per-capture exposure cancels, then inverted) and applied as a 3×3 unmix of the **linear** capture, ahead of the log/inversion where the film-dye crosstalk below lives. The related **narrowband scan** toggle instead applies the bundled RGBScan *input* profile at the display/export boundary (an explicit Input ICC overrides it).
 *   **Log Conversion**: Film density is logarithmic ($D \propto \log E$). We convert the raw signal to log-space to align with the physics of the film layers:
     $$E_{log} = \log_{10}(I_{raw})$$

docs/USER_GUIDE.md

@@ -226,7 +226,7 @@ Where the frame gets its final shape: what's inside the print, and whether it si
 Corrects uneven illumination (vignetting/falloff) from your copy-stand or scanner light, using a reference shot of the bare light source.
 
 *   **Flatfield Correction**: apply the active reference to this image (enabled once a profile exists).
-*   **Reference Profile** dropdown + **Add…** / **Delete**: pick a reference image and save it as a named profile.
+*   **Reference Profile** dropdown + **Add…** / **Delete**: pick a reference image and save it as a named profile. **Add…** reads the reference once and bakes its correction into the profile, so the original reference file can then be moved, renamed or deleted without affecting your edits — the profile is self-contained (stored in NegPy's own `flatfield` folder, like sensor and crosstalk profiles).
 *   **Distortion** (-0.25 to 0.25): radial lens-distortion correction for the rig, saved with the profile. Use the film rebate as a straight-edge reference.
 
 ---

negpy/desktop/controller.py

@@ -122,7 +122,7 @@ def _autocrop_fingerprint(config: WorkspaceConfig, workspace_color_space: str) -
         int(geometry.autocrop_offset),
         round(float(geometry.autocrop_rebate_trim), 4),
         bool(flatfield.apply),
-        str(flatfield.reference_path),
+        str(flatfield.profile_id),
         round(float(flatfield.k1), 9),
         bool(config.process.linear_raw),
         bool(rgbscan.enabled),
@@ -1030,7 +1030,7 @@ def load_file(self, file_path: str, preserve_zoom: bool = False, force_detect: b
                 stitch_transforms=stitch.stitch_transforms if stitch.stitch_enabled else (),
                 stitch_canvas=stitch.stitch_canvas,
                 stitch_sizes=stitch.stitch_sizes,
-                flatfield_path=flatfield.reference_path if (stitch.stitch_enabled and flatfield.apply) else "",
+                flatfield_profile_id=flatfield.profile_id if (stitch.stitch_enabled and flatfield.apply) else "",
             )
         )
 
@@ -2216,34 +2216,46 @@ def reanalyze_current_file(self) -> None:
         self.session.update_config(replace(self.state.config, process=new_process))
         self.request_render()
 
-    def set_active_flatfield_profile(self, name: str) -> None:
+    def set_active_flatfield_profile(self, profile_id: str) -> None:
         """
         Selects the globally active flat-field reference profile (or clears it when
-        ``name`` is empty). Applies its path to the current image and re-renders.
+        ``profile_id`` is empty). Stamps its id + rig distortion onto the current
+        image and re-renders.
         """
-        self.session.repo.save_global_setting("flatfield_active_profile", name or "")
-        rec = self.session.repo.get_flatfield_profile(name) if name else None
-        path, k1 = rec if rec else ("", 0.0)
-        new_ff = replace(self.state.config.flatfield, reference_path=path or "", apply=bool(path), k1=k1)
+        from negpy.services.assets.flatfield import FlatFieldProfiles
+
+        self.session.repo.save_global_setting("flatfield_active_profile", profile_id or "")
+        prof = FlatFieldProfiles.get(profile_id) if profile_id else None
+        pid = prof.id if prof else ""
+        new_ff = replace(self.state.config.flatfield, profile_id=pid, apply=bool(pid), k1=prof.k1 if prof else 0.0)
         self.session.update_config(replace(self.state.config, flatfield=new_ff), persist=True)
         self.request_render()
 
     def save_flatfield_profile(self, name: str, path: str) -> None:
         """
-        Saves a reference image as a named flat-field profile and makes it active.
+        Bakes a reference image into a named flat-field profile and makes it active.
         """
-        self.session.repo.save_flatfield_profile(name, path)
-        self.set_active_flatfield_profile(name)
+        from negpy.services.assets.flatfield import FlatFieldProfiles
+
+        profile_id = FlatFieldProfiles.create(name, path)
+        if profile_id is None:
+            self.set_status("Flat-field: could not read that reference image", 3000)
+            return
+        self.set_active_flatfield_profile(profile_id)
         self.set_status(f"Flat-field profile '{name}' saved", 2000)
 
-    def delete_flatfield_profile(self, name: str) -> None:
+    def delete_flatfield_profile(self, profile_id: str) -> None:
         """
         Removes a flat-field profile; clears the active correction if it was selected.
         """
-        if not name:
+        from negpy.features.flatfield.logic import invalidate_gain
+        from negpy.services.assets.flatfield import FlatFieldProfiles
+
+        if not profile_id:
             return
-        self.session.repo.delete_flatfield_profile(name)
-        if self.session.repo.get_global_setting("flatfield_active_profile") == name:
+        FlatFieldProfiles.delete(profile_id)
+        invalidate_gain(profile_id)
+        if self.session.repo.get_global_setting("flatfield_active_profile") == profile_id:
             self.set_active_flatfield_profile("")
 
     def load_gear_library(self):
@@ -2273,9 +2285,9 @@ def set_flatfield_k1(self, k1: float) -> None:
         self.session.update_config(replace(self.state.config, flatfield=new_ff), persist=True)
         active = self.session.repo.get_global_setting("flatfield_active_profile") or ""
         if active:
-            rec = self.session.repo.get_flatfield_profile(active)
-            path = rec[0] if rec else ""
-            self.session.repo.save_flatfield_profile(active, path, k1)
+            from negpy.services.assets.flatfield import FlatFieldProfiles
+
+            FlatFieldProfiles.set_k1(active, k1)
         self.request_render()
 
     # ── Scanner integration ───────────────────────────────────────────

negpy/desktop/main.py

@@ -8,8 +8,11 @@
 from negpy.desktop.controller import AppController
 from negpy.desktop.session import DesktopSessionManager
 from negpy.desktop.view.main_window import MainWindow
+from negpy.features.flatfield.logic import set_gain_provider
 from negpy.infrastructure.storage.repository import StorageRepository
 from negpy.services.assets.crosstalk import CrosstalkProfiles
+from negpy.services.assets.flatfield import FlatFieldProfiles
+from negpy.services.assets.flatfield_migration import migrate_legacy_flatfield_profiles
 from negpy.services.assets.gear import GearProfiles
 from negpy.kernel.system.config import APP_CONFIG, BASE_USER_DIR
 from negpy.kernel.system.logging import get_logger, setup_logging
@@ -129,6 +132,12 @@ def main() -> None:
         repo = StorageRepository(APP_CONFIG.edits_db_path, APP_CONFIG.settings_db_path)
         repo.initialize()
 
+        # Resolve flat-field gains by profile id from the on-disk store (keeps the
+        # numpy logic layer free of any storage dependency), then migrate any legacy
+        # DB-backed profiles into that store.
+        set_gain_provider(FlatFieldProfiles.load_gain)
+        migrate_legacy_flatfield_profiles(repo)
+
         scale = float(repo.get_global_setting("ui_scale", 1.0) or 1.0)
         scale = max(0.8, min(1.2, scale))
         if scale != 1.0 and "QT_SCALE_FACTOR" not in os.environ:

negpy/desktop/session.py

@@ -16,6 +16,7 @@
 from negpy.infrastructure.display.color_spaces import WORKING_COLOR_SPACE
 from negpy.infrastructure.storage.repository import StorageRepository
 from negpy.kernel.system.config import APP_CONFIG
+from negpy.services.assets.flatfield import FlatFieldProfiles
 from negpy.services.assets.sidecar import load_or_promote
 
 
@@ -536,13 +537,14 @@ def _apply_sticky_settings(self, config: WorkspaceConfig, only_global: bool = Fa
                 metadata=replace(config.metadata, protect_original_metadata=bool(sticky_protect)),
             )
 
-        # Flat-field reference and distortion k1 are rig-global: the active profile's
+        # Flat-field profile and distortion k1 are rig-global: the active profile's
         # values always override the per-file ones. New files default to enabled when a
         # profile is active; saved files keep their toggle.
         active_ff = self.repo.get_global_setting("flatfield_active_profile")
-        ff_rec = self.repo.get_flatfield_profile(active_ff) if active_ff else None
-        ff_path, ff_k1 = ff_rec if ff_rec else ("", 0.0)
-        config = replace(config, flatfield=replace(config.flatfield, reference_path=ff_path, k1=ff_k1))
+        ff_prof = FlatFieldProfiles.get(active_ff) if active_ff else None
+        ff_id = ff_prof.id if ff_prof else ""
+        ff_k1 = ff_prof.k1 if ff_prof else 0.0
+        config = replace(config, flatfield=replace(config.flatfield, profile_id=ff_id, k1=ff_k1))
 
         # Temperature roll-locks (per region): re-aim each locked region's M/Y
         # pair at its Kelvin target, keeping the frame's own off-locus tint.
@@ -561,7 +563,7 @@ def _apply_sticky_settings(self, config: WorkspaceConfig, only_global: bool = Fa
         if only_global:
             return config
 
-        config = replace(config, flatfield=replace(config.flatfield, apply=bool(ff_path)))
+        config = replace(config, flatfield=replace(config.flatfield, apply=bool(ff_id)))
 
         # Workflow settings — safe to carry across all files on a roll
         sticky_mode = self.repo.get_global_setting("last_process_mode")

negpy/desktop/view/sidebar/flatfield.py

@@ -1,7 +1,9 @@
 import os
 
 import qtawesome as qta
+from PyQt6.QtCore import Qt
 from PyQt6.QtWidgets import (
+    QApplication,
     QComboBox,
     QFileDialog,
     QHBoxLayout,
@@ -78,20 +80,22 @@ def _connect_signals(self) -> None:
     def _refresh_profiles(self) -> None:
         # Preserve the caller's block state: unblocking here would let sync_ui's
         # setCurrentIndex re-fire _on_profile_selected and loop into update_config.
+        from negpy.services.assets.flatfield import FlatFieldProfiles
+
         prev = self.profile_combo.signalsBlocked()
         self.profile_combo.blockSignals(True)
         self.profile_combo.clear()
         self.profile_combo.addItem(_NONE_LABEL, "")
-        for name in self.controller.session.repo.list_flatfield_profiles():
-            self.profile_combo.addItem(name, name)
+        for profile_id, name in FlatFieldProfiles.list_profiles():
+            self.profile_combo.addItem(name, profile_id)
         self.profile_combo.blockSignals(prev)
 
     def _on_profile_selected(self, _idx: int) -> None:
-        name = self.profile_combo.currentData() or ""
+        profile_id = self.profile_combo.currentData() or ""
         active = self.controller.session.repo.get_global_setting("flatfield_active_profile") or ""
-        if name == active:
+        if profile_id == active:
             return
-        self.controller.set_active_flatfield_profile(name)
+        self.controller.set_active_flatfield_profile(profile_id)
         self.sync_ui()
 
     def _on_add(self) -> None:
@@ -101,14 +105,20 @@ def _on_add(self) -> None:
         default_name = os.path.splitext(os.path.basename(path))[0]
         name, ok = QInputDialog.getText(self, "Save Flat-Field Profile", "Profile name:", text=default_name)
         if ok and name:
-            self.controller.save_flatfield_profile(name, path)
+            # save_flatfield_profile decodes the reference RAW to bake the gain — a
+            # brief blocking beat on the GUI thread, so show a wait cursor.
+            QApplication.setOverrideCursor(Qt.CursorShape.WaitCursor)
+            try:
+                self.controller.save_flatfield_profile(name, path)
+            finally:
+                QApplication.restoreOverrideCursor()
             self._refresh_profiles()
             self.sync_ui()
 
     def _on_delete(self) -> None:
-        name = self.profile_combo.currentData()
-        if name:
-            self.controller.delete_flatfield_profile(name)
+        profile_id = self.profile_combo.currentData()
+        if profile_id:
+            self.controller.delete_flatfield_profile(profile_id)
             self._refresh_profiles()
             self.sync_ui()
 
@@ -123,9 +133,9 @@ def sync_ui(self) -> None:
             self.profile_combo.setCurrentIndex(idx if idx >= 0 else 0)
 
             self.enable_btn.setChecked(conf.apply)
-            self.enable_btn.setEnabled(bool(conf.reference_path))
+            self.enable_btn.setEnabled(bool(conf.profile_id))
             self.k1_slider.setValue(conf.k1)
-            self.k1_slider.setEnabled(bool(conf.reference_path))
+            self.k1_slider.setEnabled(bool(conf.profile_id))
         finally:
             self.block_signals(False)
 

negpy/desktop/view/widgets/database_dialog.py

@@ -24,7 +24,6 @@
 )
 _TOOLING_ROWS = (
     ("normalization_rolls", "Normalization rolls"),
-    ("flatfield_profiles", "Flat-field profiles"),
     ("export_presets", "Export presets"),
     ("app_preferences", "App preferences"),
 )
@@ -173,7 +172,7 @@ def _refresh(self) -> None:
 
     def _update_enabled(self, stats: dict) -> None:
         edits = stats.get("file_settings", 0) + stats.get("edit_history", 0) + stats.get("file_marks", 0)
-        total = edits + sum(stats.get(k, 0) for k in ("normalization_rolls", "flatfield_profiles", "export_presets", "app_preferences"))
+        total = edits + sum(stats.get(k, 0) for k in ("normalization_rolls", "export_presets", "app_preferences"))
         self.clear_edits_btn.setEnabled(edits > 0)
         self.reset_all_btn.setEnabled(total > 0)
         self.clear_thumbs_btn.setEnabled(stats.get("thumbnails", 0) > 0)

negpy/desktop/workers/render.py

@@ -165,7 +165,7 @@ class PreviewLoadTask:
     stitch_transforms: tuple[tuple[float, ...], ...] = ()
     stitch_canvas: tuple[int, int] = (0, 0)
     stitch_sizes: tuple[tuple[int, int], ...] = ()
-    flatfield_path: str = ""  # per-part flat-field for stitch previews
+    flatfield_profile_id: str = ""  # per-part flat-field profile for stitch previews
 
 
 class RenderWorker(QObject):
@@ -585,7 +585,7 @@ def process(self, task: PreviewLoadTask) -> None:
                     use_camera_wb=task.use_camera_wb,
                     full_resolution=task.full_resolution,
                     file_hash=task.file_hash,
-                    flatfield_path=task.flatfield_path,
+                    flatfield_profile_id=task.flatfield_profile_id,
                 )
                 source_cs = metadata.get("color_space") or WORKING_COLOR_SPACE
                 ir_preview = metadata.get("ir_preview")

negpy/domain/migrations.py

@@ -47,6 +47,11 @@
         "density_saturation_damping",
         "density_damping_spatial",
         "vibrance",  # Lab Vibrance, retired in favour of the per-pixel Dye Separation
+        # Flat-field moved from a stored reference *path* to an opaque profile_id
+        # (baked gain in the file store). The DB migration rewrites edits it can
+        # reach; this drops the legacy key from any that slip through (sidecars,
+        # presets) so it doesn't warn — that edit just needs its profile re-picked.
+        "reference_path",
     }
 )
 

negpy/domain/types.py

@@ -36,6 +36,7 @@ class AppConfig:
     user_icc_dir: str
     crosstalk_dir: str
     sensor_dir: str
+    flatfield_dir: str
     gear_dir: str
     contact_sheet_templates_dir: str
     default_export_dir: str