diff --git a/CLAUDE.md b/CLAUDE.md index dba6bd6c..c2f7fcfa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,3 +81,4 @@ Every feature lives in `negpy/features//`: - **CPU/GPU parity**: any change to a stage's math must land in both `logic.py` and its `.wgsl` shader. Constants mirrored as WGSL literals (histogram bins, zone density, metrics offsets) have parity tests — keep them in sync. - **Working-space OETF + luminance row are inlined** in `lab_sharpen_h.wgsl` and `rl_init.wgsl` (Adobe RGB 1998 gamma 563/256, D65 Y row) — a TRC or primaries change must update them, not just `kernel/image/logic.py`. `LabUniforms` is declared in 6 lab shaders (lab, lab_sharpen_h/v, rl_blur_h, rl_div_v, rl_mult_v) — any field change touches all six plus the `struct.pack` in `gpu_engine.py`, and the trailing `_pad*` floats keep the block at 48 bytes. `rl_init.wgsl` binds no uniform at all (the auto layout prunes it). +- **Flat-field gains resolve through a provider, not the config.** The per-image `FlatFieldConfig` carries only an opaque `profile_id`; the baked gain map lives in a per-profile `.npz` in `APP_CONFIG.flatfield_dir` (`services/assets/flatfield.py`, the sensor/crosstalk file-store pattern). `apply_flatfield`/`flatfield_token` look the gain up via `set_gain_provider`, which `desktop/main.py` wires at startup — any render path outside the desktop app (a script, a headless test exercising flat-field) must call `set_gain_provider` first or the correction silently no-ops. Legacy DB profiles (the retired `flatfield_profiles` table) are one-shot migrated by `flatfield_migration.py`. diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 139b9226..fbcea3ce 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -21,7 +21,7 @@ Here is what actually happens to your image. We apply these steps in order, pass * **Physical Model**: We treat the input as a **radiometric measurement**. Pixel values represent linear transmittance captured by the sensor. * **Source corrections** (linear domain, before the log conversion): - * **Flat-field** (`negpy.features.flatfield`): divides out illumination falloff using a blank reference frame. A per-channel gain map $\text{mean}(\text{blur})/\text{blur}$ (computed on a 256 px copy, clamped to $[0.25, 4]$) multiplies the linear source. The reference's identity is folded into the render's source hash. + * **Flat-field** (`negpy.features.flatfield`): divides out illumination falloff using a blank reference frame. A per-channel gain map $\text{mean}(\text{blur})/\text{blur}$ (computed on a 256 px copy, clamped to $[0.25, 4]$) multiplies the linear source. The reference is decoded and the gain **baked once** into a profile (an `.npz` in `APP_CONFIG.flatfield_dir`, keyed by an opaque id), so the render never touches the original reference file — moving or deleting it is harmless. The per-image edit stores only the profile id; the render path resolves the gain through a provider (`set_gain_provider`, wired to `services/assets/flatfield.py` at startup) and caches it, and the profile id + a content token of the gain are folded into the render's source hash. * **Sensor crosstalk unmix** (`sensor_matrix`, `features/process/sensor.py`): for single-shot narrowband camera scans, the camera's CFA passbands overlap the light source's bands, so a pure R/G/B exposure leaks into the other channels. That is a fixed property of the sensor+light pair, independent of film. It is calibrated once from three bare-light exposures (response columns normalized to a unit diagonal so per-capture exposure cancels, then inverted) and applied as a 3×3 unmix of the **linear** capture, ahead of the log/inversion where the film-dye crosstalk below lives. The related **narrowband scan** toggle instead applies the bundled RGBScan *input* profile at the display/export boundary (an explicit Input ICC overrides it). * **Log Conversion**: Film density is logarithmic ($D \propto \log E$). We convert the raw signal to log-space to align with the physics of the film layers: $$E_{log} = \log_{10}(I_{raw})$$ diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index 82637db9..96ffc5a4 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -226,7 +226,7 @@ Where the frame gets its final shape: what's inside the print, and whether it si Corrects uneven illumination (vignetting/falloff) from your copy-stand or scanner light, using a reference shot of the bare light source. * **Flatfield Correction**: apply the active reference to this image (enabled once a profile exists). -* **Reference Profile** dropdown + **Add…** / **Delete**: pick a reference image and save it as a named profile. +* **Reference Profile** dropdown + **Add…** / **Delete**: pick a reference image and save it as a named profile. **Add…** reads the reference once and bakes its correction into the profile, so the original reference file can then be moved, renamed or deleted without affecting your edits — the profile is self-contained (stored in NegPy's own `flatfield` folder, like sensor and crosstalk profiles). * **Distortion** (-0.25 to 0.25): radial lens-distortion correction for the rig, saved with the profile. Use the film rebate as a straight-edge reference. --- diff --git a/negpy/desktop/controller.py b/negpy/desktop/controller.py index 053ea14a..d62a05b6 100644 --- a/negpy/desktop/controller.py +++ b/negpy/desktop/controller.py @@ -122,7 +122,7 @@ def _autocrop_fingerprint(config: WorkspaceConfig, workspace_color_space: str) - int(geometry.autocrop_offset), round(float(geometry.autocrop_rebate_trim), 4), bool(flatfield.apply), - str(flatfield.reference_path), + str(flatfield.profile_id), round(float(flatfield.k1), 9), bool(config.process.linear_raw), bool(rgbscan.enabled), @@ -1030,7 +1030,7 @@ def load_file(self, file_path: str, preserve_zoom: bool = False, force_detect: b stitch_transforms=stitch.stitch_transforms if stitch.stitch_enabled else (), stitch_canvas=stitch.stitch_canvas, stitch_sizes=stitch.stitch_sizes, - flatfield_path=flatfield.reference_path if (stitch.stitch_enabled and flatfield.apply) else "", + flatfield_profile_id=flatfield.profile_id if (stitch.stitch_enabled and flatfield.apply) else "", ) ) @@ -2216,34 +2216,46 @@ def reanalyze_current_file(self) -> None: self.session.update_config(replace(self.state.config, process=new_process)) self.request_render() - def set_active_flatfield_profile(self, name: str) -> None: + def set_active_flatfield_profile(self, profile_id: str) -> None: """ Selects the globally active flat-field reference profile (or clears it when - ``name`` is empty). Applies its path to the current image and re-renders. + ``profile_id`` is empty). Stamps its id + rig distortion onto the current + image and re-renders. """ - self.session.repo.save_global_setting("flatfield_active_profile", name or "") - rec = self.session.repo.get_flatfield_profile(name) if name else None - path, k1 = rec if rec else ("", 0.0) - new_ff = replace(self.state.config.flatfield, reference_path=path or "", apply=bool(path), k1=k1) + from negpy.services.assets.flatfield import FlatFieldProfiles + + self.session.repo.save_global_setting("flatfield_active_profile", profile_id or "") + prof = FlatFieldProfiles.get(profile_id) if profile_id else None + pid = prof.id if prof else "" + new_ff = replace(self.state.config.flatfield, profile_id=pid, apply=bool(pid), k1=prof.k1 if prof else 0.0) self.session.update_config(replace(self.state.config, flatfield=new_ff), persist=True) self.request_render() def save_flatfield_profile(self, name: str, path: str) -> None: """ - Saves a reference image as a named flat-field profile and makes it active. + Bakes a reference image into a named flat-field profile and makes it active. """ - self.session.repo.save_flatfield_profile(name, path) - self.set_active_flatfield_profile(name) + from negpy.services.assets.flatfield import FlatFieldProfiles + + profile_id = FlatFieldProfiles.create(name, path) + if profile_id is None: + self.set_status("Flat-field: could not read that reference image", 3000) + return + self.set_active_flatfield_profile(profile_id) self.set_status(f"Flat-field profile '{name}' saved", 2000) - def delete_flatfield_profile(self, name: str) -> None: + def delete_flatfield_profile(self, profile_id: str) -> None: """ Removes a flat-field profile; clears the active correction if it was selected. """ - if not name: + from negpy.features.flatfield.logic import invalidate_gain + from negpy.services.assets.flatfield import FlatFieldProfiles + + if not profile_id: return - self.session.repo.delete_flatfield_profile(name) - if self.session.repo.get_global_setting("flatfield_active_profile") == name: + FlatFieldProfiles.delete(profile_id) + invalidate_gain(profile_id) + if self.session.repo.get_global_setting("flatfield_active_profile") == profile_id: self.set_active_flatfield_profile("") def load_gear_library(self): @@ -2273,9 +2285,9 @@ def set_flatfield_k1(self, k1: float) -> None: self.session.update_config(replace(self.state.config, flatfield=new_ff), persist=True) active = self.session.repo.get_global_setting("flatfield_active_profile") or "" if active: - rec = self.session.repo.get_flatfield_profile(active) - path = rec[0] if rec else "" - self.session.repo.save_flatfield_profile(active, path, k1) + from negpy.services.assets.flatfield import FlatFieldProfiles + + FlatFieldProfiles.set_k1(active, k1) self.request_render() # ── Scanner integration ─────────────────────────────────────────── diff --git a/negpy/desktop/main.py b/negpy/desktop/main.py index 4e54e97d..dde4e6d9 100644 --- a/negpy/desktop/main.py +++ b/negpy/desktop/main.py @@ -8,8 +8,11 @@ from negpy.desktop.controller import AppController from negpy.desktop.session import DesktopSessionManager from negpy.desktop.view.main_window import MainWindow +from negpy.features.flatfield.logic import set_gain_provider from negpy.infrastructure.storage.repository import StorageRepository from negpy.services.assets.crosstalk import CrosstalkProfiles +from negpy.services.assets.flatfield import FlatFieldProfiles +from negpy.services.assets.flatfield_migration import migrate_legacy_flatfield_profiles from negpy.services.assets.gear import GearProfiles from negpy.kernel.system.config import APP_CONFIG, BASE_USER_DIR from negpy.kernel.system.logging import get_logger, setup_logging @@ -129,6 +132,12 @@ def main() -> None: repo = StorageRepository(APP_CONFIG.edits_db_path, APP_CONFIG.settings_db_path) repo.initialize() + # Resolve flat-field gains by profile id from the on-disk store (keeps the + # numpy logic layer free of any storage dependency), then migrate any legacy + # DB-backed profiles into that store. + set_gain_provider(FlatFieldProfiles.load_gain) + migrate_legacy_flatfield_profiles(repo) + scale = float(repo.get_global_setting("ui_scale", 1.0) or 1.0) scale = max(0.8, min(1.2, scale)) if scale != 1.0 and "QT_SCALE_FACTOR" not in os.environ: diff --git a/negpy/desktop/session.py b/negpy/desktop/session.py index d863d888..a7840edb 100644 --- a/negpy/desktop/session.py +++ b/negpy/desktop/session.py @@ -16,6 +16,7 @@ from negpy.infrastructure.display.color_spaces import WORKING_COLOR_SPACE from negpy.infrastructure.storage.repository import StorageRepository from negpy.kernel.system.config import APP_CONFIG +from negpy.services.assets.flatfield import FlatFieldProfiles from negpy.services.assets.sidecar import load_or_promote @@ -536,13 +537,14 @@ def _apply_sticky_settings(self, config: WorkspaceConfig, only_global: bool = Fa metadata=replace(config.metadata, protect_original_metadata=bool(sticky_protect)), ) - # Flat-field reference and distortion k1 are rig-global: the active profile's + # Flat-field profile and distortion k1 are rig-global: the active profile's # values always override the per-file ones. New files default to enabled when a # profile is active; saved files keep their toggle. active_ff = self.repo.get_global_setting("flatfield_active_profile") - ff_rec = self.repo.get_flatfield_profile(active_ff) if active_ff else None - ff_path, ff_k1 = ff_rec if ff_rec else ("", 0.0) - config = replace(config, flatfield=replace(config.flatfield, reference_path=ff_path, k1=ff_k1)) + ff_prof = FlatFieldProfiles.get(active_ff) if active_ff else None + ff_id = ff_prof.id if ff_prof else "" + ff_k1 = ff_prof.k1 if ff_prof else 0.0 + config = replace(config, flatfield=replace(config.flatfield, profile_id=ff_id, k1=ff_k1)) # Temperature roll-locks (per region): re-aim each locked region's M/Y # pair at its Kelvin target, keeping the frame's own off-locus tint. @@ -561,7 +563,7 @@ def _apply_sticky_settings(self, config: WorkspaceConfig, only_global: bool = Fa if only_global: return config - config = replace(config, flatfield=replace(config.flatfield, apply=bool(ff_path))) + config = replace(config, flatfield=replace(config.flatfield, apply=bool(ff_id))) # Workflow settings — safe to carry across all files on a roll sticky_mode = self.repo.get_global_setting("last_process_mode") diff --git a/negpy/desktop/view/sidebar/flatfield.py b/negpy/desktop/view/sidebar/flatfield.py index 5d264143..02bbc2c0 100644 --- a/negpy/desktop/view/sidebar/flatfield.py +++ b/negpy/desktop/view/sidebar/flatfield.py @@ -1,7 +1,9 @@ import os import qtawesome as qta +from PyQt6.QtCore import Qt from PyQt6.QtWidgets import ( + QApplication, QComboBox, QFileDialog, QHBoxLayout, @@ -78,20 +80,22 @@ def _connect_signals(self) -> None: def _refresh_profiles(self) -> None: # Preserve the caller's block state: unblocking here would let sync_ui's # setCurrentIndex re-fire _on_profile_selected and loop into update_config. + from negpy.services.assets.flatfield import FlatFieldProfiles + prev = self.profile_combo.signalsBlocked() self.profile_combo.blockSignals(True) self.profile_combo.clear() self.profile_combo.addItem(_NONE_LABEL, "") - for name in self.controller.session.repo.list_flatfield_profiles(): - self.profile_combo.addItem(name, name) + for profile_id, name in FlatFieldProfiles.list_profiles(): + self.profile_combo.addItem(name, profile_id) self.profile_combo.blockSignals(prev) def _on_profile_selected(self, _idx: int) -> None: - name = self.profile_combo.currentData() or "" + profile_id = self.profile_combo.currentData() or "" active = self.controller.session.repo.get_global_setting("flatfield_active_profile") or "" - if name == active: + if profile_id == active: return - self.controller.set_active_flatfield_profile(name) + self.controller.set_active_flatfield_profile(profile_id) self.sync_ui() def _on_add(self) -> None: @@ -101,14 +105,20 @@ def _on_add(self) -> None: default_name = os.path.splitext(os.path.basename(path))[0] name, ok = QInputDialog.getText(self, "Save Flat-Field Profile", "Profile name:", text=default_name) if ok and name: - self.controller.save_flatfield_profile(name, path) + # save_flatfield_profile decodes the reference RAW to bake the gain — a + # brief blocking beat on the GUI thread, so show a wait cursor. + QApplication.setOverrideCursor(Qt.CursorShape.WaitCursor) + try: + self.controller.save_flatfield_profile(name, path) + finally: + QApplication.restoreOverrideCursor() self._refresh_profiles() self.sync_ui() def _on_delete(self) -> None: - name = self.profile_combo.currentData() - if name: - self.controller.delete_flatfield_profile(name) + profile_id = self.profile_combo.currentData() + if profile_id: + self.controller.delete_flatfield_profile(profile_id) self._refresh_profiles() self.sync_ui() @@ -123,9 +133,9 @@ def sync_ui(self) -> None: self.profile_combo.setCurrentIndex(idx if idx >= 0 else 0) self.enable_btn.setChecked(conf.apply) - self.enable_btn.setEnabled(bool(conf.reference_path)) + self.enable_btn.setEnabled(bool(conf.profile_id)) self.k1_slider.setValue(conf.k1) - self.k1_slider.setEnabled(bool(conf.reference_path)) + self.k1_slider.setEnabled(bool(conf.profile_id)) finally: self.block_signals(False) diff --git a/negpy/desktop/view/widgets/database_dialog.py b/negpy/desktop/view/widgets/database_dialog.py index 03773b9d..b6a970b8 100644 --- a/negpy/desktop/view/widgets/database_dialog.py +++ b/negpy/desktop/view/widgets/database_dialog.py @@ -24,7 +24,6 @@ ) _TOOLING_ROWS = ( ("normalization_rolls", "Normalization rolls"), - ("flatfield_profiles", "Flat-field profiles"), ("export_presets", "Export presets"), ("app_preferences", "App preferences"), ) @@ -173,7 +172,7 @@ def _refresh(self) -> None: def _update_enabled(self, stats: dict) -> None: edits = stats.get("file_settings", 0) + stats.get("edit_history", 0) + stats.get("file_marks", 0) - total = edits + sum(stats.get(k, 0) for k in ("normalization_rolls", "flatfield_profiles", "export_presets", "app_preferences")) + total = edits + sum(stats.get(k, 0) for k in ("normalization_rolls", "export_presets", "app_preferences")) self.clear_edits_btn.setEnabled(edits > 0) self.reset_all_btn.setEnabled(total > 0) self.clear_thumbs_btn.setEnabled(stats.get("thumbnails", 0) > 0) diff --git a/negpy/desktop/workers/render.py b/negpy/desktop/workers/render.py index d8deb8cf..29ed6043 100644 --- a/negpy/desktop/workers/render.py +++ b/negpy/desktop/workers/render.py @@ -165,7 +165,7 @@ class PreviewLoadTask: stitch_transforms: tuple[tuple[float, ...], ...] = () stitch_canvas: tuple[int, int] = (0, 0) stitch_sizes: tuple[tuple[int, int], ...] = () - flatfield_path: str = "" # per-part flat-field for stitch previews + flatfield_profile_id: str = "" # per-part flat-field profile for stitch previews class RenderWorker(QObject): @@ -585,7 +585,7 @@ def process(self, task: PreviewLoadTask) -> None: use_camera_wb=task.use_camera_wb, full_resolution=task.full_resolution, file_hash=task.file_hash, - flatfield_path=task.flatfield_path, + flatfield_profile_id=task.flatfield_profile_id, ) source_cs = metadata.get("color_space") or WORKING_COLOR_SPACE ir_preview = metadata.get("ir_preview") diff --git a/negpy/domain/migrations.py b/negpy/domain/migrations.py index 64c8b0bd..b61073f0 100644 --- a/negpy/domain/migrations.py +++ b/negpy/domain/migrations.py @@ -47,6 +47,11 @@ "density_saturation_damping", "density_damping_spatial", "vibrance", # Lab Vibrance, retired in favour of the per-pixel Dye Separation + # Flat-field moved from a stored reference *path* to an opaque profile_id + # (baked gain in the file store). The DB migration rewrites edits it can + # reach; this drops the legacy key from any that slip through (sidecars, + # presets) so it doesn't warn — that edit just needs its profile re-picked. + "reference_path", } ) diff --git a/negpy/domain/types.py b/negpy/domain/types.py index 46388382..c63cf5c2 100644 --- a/negpy/domain/types.py +++ b/negpy/domain/types.py @@ -36,6 +36,7 @@ class AppConfig: user_icc_dir: str crosstalk_dir: str sensor_dir: str + flatfield_dir: str gear_dir: str contact_sheet_templates_dir: str default_export_dir: str diff --git a/negpy/features/flatfield/logic.py b/negpy/features/flatfield/logic.py index 7eae21b4..2fb55774 100644 --- a/negpy/features/flatfield/logic.py +++ b/negpy/features/flatfield/logic.py @@ -1,29 +1,56 @@ -import os -from typing import Dict, Optional, Tuple +import hashlib +from typing import Callable, Dict, Optional, Tuple import cv2 import numpy as np from negpy.domain.types import ImageBuffer from negpy.features.flatfield.models import FlatFieldConfig -from negpy.kernel.system.logging import get_logger - -logger = get_logger(__name__) - -# Gain maps cached by (path, mtime); decoding the reference is slow. -_GAIN_CACHE: Dict[Tuple[str, float], Optional[np.ndarray]] = {} # Clamp so a near-black reference pixel can't blow up the image. _GAIN_MIN = 0.25 _GAIN_MAX = 4.0 - # Falloff is low-frequency: compute the gain on a small copy (upscaled at apply time) # so the blur kernel stays tiny. _GAIN_WORK_SIZE = 256 +# Resolved gains keyed by profile id: (gain map, content token). A cached ``None`` +# marks a known-missing profile so a broken reference doesn't re-hit the store every +# render. Populated lazily through the injected provider — the desktop app wires it +# to the on-disk profile store (services/assets/flatfield.py) at startup; tests may +# seed this map directly. +GainEntry = Tuple[np.ndarray, str] +_GAIN_CACHE: Dict[str, Optional[GainEntry]] = {} +_gain_provider: Optional[Callable[[str], Optional[GainEntry]]] = None + + +def set_gain_provider(provider: Optional[Callable[[str], Optional[GainEntry]]]) -> None: + """Inject the ``profile_id -> (gain, token)`` resolver and drop any cached gains.""" + global _gain_provider + _gain_provider = provider + _GAIN_CACHE.clear() -def _compute_gain(reference: ImageBuffer) -> np.ndarray: + +def invalidate_gain(profile_id: Optional[str] = None) -> None: + """Drop a cached gain (all when None) after a profile is re-baked or deleted.""" + if profile_id is None: + _GAIN_CACHE.clear() + else: + _GAIN_CACHE.pop(profile_id, None) + + +def _resolve(profile_id: str) -> Optional[GainEntry]: + if not profile_id: + return None + if profile_id in _GAIN_CACHE: + return _GAIN_CACHE[profile_id] + entry = _gain_provider(profile_id) if _gain_provider is not None else None + _GAIN_CACHE[profile_id] = entry + return entry + + +def compute_gain(reference: ImageBuffer) -> np.ndarray: """Per-channel gain = mean(blur) / blur, on a downsampled copy.""" ref = reference.astype(np.float32) h, w = ref.shape[:2] @@ -39,51 +66,29 @@ def _compute_gain(reference: ImageBuffer) -> np.ndarray: return np.clip(gain, _GAIN_MIN, _GAIN_MAX).astype(np.float32) -def load_reference_gain(path: str) -> Optional[np.ndarray]: - """Per-channel gain map for the reference, decoded like a negative (no WB, linear).""" - if not path or not os.path.exists(path): - return None - try: - mtime = os.path.getmtime(path) - except OSError: - return None - - key = (path, mtime) - if key in _GAIN_CACHE: - return _GAIN_CACHE[key] - - gain: Optional[np.ndarray] = None - try: - from negpy.services.rendering.preview_manager import PreviewManager - - reference, _, _ = PreviewManager().load_linear_preview(path, use_camera_wb=False, full_resolution=False) - gain = _compute_gain(reference) - except Exception: - logger.exception("Flat-field: failed to load reference %s", path) - gain = None - - _GAIN_CACHE[key] = gain - return gain +def gain_token(gain: np.ndarray) -> str: + """Stable content id for a baked gain map, folded into the render source hash.""" + return hashlib.blake2b(np.ascontiguousarray(gain, dtype=np.float32).tobytes(), digest_size=8).hexdigest() def flatfield_token(config: FlatFieldConfig) -> str: """Identity of the active correction, folded into the render source hash. Empty when inactive.""" - if not config.apply or not config.reference_path: + if not config.apply or not config.profile_id: return "" - try: - mtime = os.path.getmtime(config.reference_path) - except OSError: + entry = _resolve(config.profile_id) + if entry is None: return "" - return f"|ff:{config.reference_path}:{mtime}" + return f"|ff:{config.profile_id}:{entry[1]}" def apply_flatfield(image: ImageBuffer, config: FlatFieldConfig) -> ImageBuffer: - """Multiply the linear source by the reference gain map. No-op when inactive or unreadable.""" - if not config.apply or not config.reference_path: + """Multiply the linear source by the reference gain map. No-op when inactive or unresolved.""" + if not config.apply or not config.profile_id: return image - gain = load_reference_gain(config.reference_path) - if gain is None: + entry = _resolve(config.profile_id) + if entry is None: return image + gain = entry[0] if gain.shape[:2] != image.shape[:2]: gain = cv2.resize(gain, (image.shape[1], image.shape[0]), interpolation=cv2.INTER_LINEAR) return (image * gain).astype(np.float32) diff --git a/negpy/features/flatfield/models.py b/negpy/features/flatfield/models.py index ac2cd27f..586f1bd1 100644 --- a/negpy/features/flatfield/models.py +++ b/negpy/features/flatfield/models.py @@ -8,8 +8,12 @@ class FlatFieldConfig: # Per-image toggle. Named 'apply', not 'enabled', to stay unique in the flat # config dict (WorkspaceConfig.to_dict) where RgbScanConfig.enabled also lives. apply: bool = False - # Resolved path of the globally active reference profile (seeded on file load). - reference_path: str = "" + # Opaque id of the active reference profile — a baked gain map stored in + # APP_CONFIG.flatfield_dir (services/assets/flatfield.py). Stable across renames + # and machines; the render path resolves the gain by this id, so the original + # reference image can be moved or deleted without breaking the correction. + profile_id: str = "" # Radial lens-distortion coefficient. A rig property, so it's mirrored from the - # active profile (re-seeded on load), not owned by the per-image edit. + # active profile (re-seeded on load) and consumed by the geometry stage — not + # owned by the per-image edit. k1: float = 0.0 diff --git a/negpy/infrastructure/storage/repository.py b/negpy/infrastructure/storage/repository.py index b2ad9ebb..4dc65ca8 100644 --- a/negpy/infrastructure/storage/repository.py +++ b/negpy/infrastructure/storage/repository.py @@ -55,18 +55,6 @@ def initialize(self) -> None: except sqlite3.OperationalError: pass - conn.execute(""" - CREATE TABLE IF NOT EXISTS flatfield_profiles ( - name TEXT PRIMARY KEY, - path TEXT - ) - """) - # Migration: add radial distortion coefficient (rig-level, like the flat frame). - try: - conn.execute("ALTER TABLE flatfield_profiles ADD COLUMN k1 REAL DEFAULT 0.0") - except sqlite3.OperationalError: - pass - conn.execute(""" CREATE TABLE IF NOT EXISTS edit_history ( file_hash TEXT, @@ -142,34 +130,6 @@ def delete_normalization_roll(self, name: str) -> None: with self._connect(self.edits_db_path) as conn: conn.execute("DELETE FROM normalization_rolls WHERE name = ?", (name,)) - def save_flatfield_profile(self, name: str, path: str, k1: float = 0.0) -> None: - """Persists a named flat-field reference profile (reference path + rig distortion).""" - with self._connect(self.edits_db_path) as conn: - conn.execute( - "INSERT OR REPLACE INTO flatfield_profiles (name, path, k1) VALUES (?, ?, ?)", - (name, path, float(k1)), - ) - - def get_flatfield_profile(self, name: str) -> Optional[tuple[str, float]]: - """Returns (reference path, distortion k1) for a named flat-field profile.""" - with self._connect(self.edits_db_path) as conn: - cursor = conn.execute("SELECT path, k1 FROM flatfield_profiles WHERE name = ?", (name,)) - row = cursor.fetchone() - if row: - return str(row[0]), float(row[1] or 0.0) - return None - - def list_flatfield_profiles(self) -> list[str]: - """Returns names of all saved flat-field profiles.""" - with self._connect(self.edits_db_path) as conn: - cursor = conn.execute("SELECT name FROM flatfield_profiles ORDER BY name") - return [row[0] for row in cursor.fetchall()] - - def delete_flatfield_profile(self, name: str) -> None: - """Deletes a named flat-field profile.""" - with self._connect(self.edits_db_path) as conn: - conn.execute("DELETE FROM flatfield_profiles WHERE name = ?", (name,)) - def save_file_mark(self, file_hash: str, mark: Optional[str]) -> None: """Persists a triage mark ('keeper'/'excluded'); None clears it.""" with self._connect(self.edits_db_path) as conn: @@ -387,7 +347,6 @@ def database_stats(self) -> dict[str, int]: edit_history = self._count(conn, "edit_history") file_marks = self._count(conn, "file_marks") normalization_rolls = self._count(conn, "normalization_rolls") - flatfield_profiles = self._count(conn, "flatfield_profiles") with self._connect(self.settings_db_path) as conn: global_settings = self._count(conn, "global_settings") @@ -401,7 +360,6 @@ def database_stats(self) -> dict[str, int]: "edit_history": edit_history, "file_marks": file_marks, "normalization_rolls": normalization_rolls, - "flatfield_profiles": flatfield_profiles, "export_presets": export_presets, # global_settings rows minus the single export_presets row (if present). "app_preferences": max(0, global_settings - (1 if has_presets_row else 0)), @@ -426,17 +384,20 @@ def _wipe(self, path: str, tables: list[str]) -> None: def clear_saved_edits(self) -> None: """Drop per-image looks: saved edits, their undo history, and keep/reject - marks. Rig calibration (normalization rolls, flat-field profiles), export - presets, and app preferences are left intact — so a reloaded image starts - from defaults without losing the user's tooling.""" + marks. Rig calibration (normalization rolls), export presets, and app + preferences are left intact — so a reloaded image starts from defaults + without losing the user's tooling. Flat-field profiles live in the file + store (APP_CONFIG.flatfield_dir), not here, so they are untouched too.""" self._wipe(self.edits_db_path, ["file_settings", "edit_history", "file_marks"]) def reset_everything(self) -> None: """Full clean slate: every table in both databases. Export presets, rig profiles, and all app preferences go too. Schema is preserved (rows only), - so the app keeps working against the emptied databases without re-init.""" + so the app keeps working against the emptied databases without re-init. + File-store assets (flat-field profiles, sensor/crosstalk matrices) are on + disk, not in these databases, so they survive — as with a fresh install.""" self._wipe( self.edits_db_path, - ["file_settings", "edit_history", "file_marks", "normalization_rolls", "flatfield_profiles"], + ["file_settings", "edit_history", "file_marks", "normalization_rolls"], ) self._wipe(self.settings_db_path, ["global_settings"]) diff --git a/negpy/kernel/system/config.py b/negpy/kernel/system/config.py index 5f7ed776..700731f8 100644 --- a/negpy/kernel/system/config.py +++ b/negpy/kernel/system/config.py @@ -34,6 +34,7 @@ user_icc_dir=os.path.join(BASE_USER_DIR, "icc"), crosstalk_dir=os.path.join(BASE_USER_DIR, "crosstalk"), sensor_dir=os.path.join(BASE_USER_DIR, "sensor"), + flatfield_dir=os.path.join(BASE_USER_DIR, "flatfield"), gear_dir=os.path.join(BASE_USER_DIR, "gear"), contact_sheet_templates_dir=os.path.join(BASE_USER_DIR, "contact_sheets"), default_export_dir=os.path.join(BASE_USER_DIR, "export"), diff --git a/negpy/services/assets/flatfield.py b/negpy/services/assets/flatfield.py new file mode 100644 index 00000000..555ac13c --- /dev/null +++ b/negpy/services/assets/flatfield.py @@ -0,0 +1,159 @@ +import os +import uuid +from typing import Dict, List, NamedTuple, Optional, Tuple + +import numpy as np + +from negpy.features.flatfield.logic import compute_gain, gain_token +from negpy.kernel.system.config import APP_CONFIG +from negpy.kernel.system.logging import get_logger + +logger = get_logger(__name__) + +_EXT = ".npz" + + +class FlatFieldProfile(NamedTuple): + id: str + name: str + k1: float + source: str # provenance path of the reference the gain was baked from + + +class FlatFieldProfiles: + """ + npz I/O for flat-field reference profiles (illumination-falloff gain maps). + + One file per profile in ``APP_CONFIG.flatfield_dir``, named ``.npz`` and + holding the baked per-channel gain map plus rig metadata: the distortion ``k1``, + a display name and the provenance path. The reference image is decoded once, at + save time; nothing outside this directory is needed to apply the correction + afterward, so moving or deleting the original reference is harmless. + + Keyed by an opaque id rather than the name — the per-image edit references a + profile by id, so a rename (or a name collision across machines) never breaks + the reference. Disk I/O happens on dropdown build, selection and save, never per + render (the render path resolves gains through the logic-layer provider cache). + """ + + @staticmethod + def _path_for_id(profile_id: str) -> str: + return os.path.join(APP_CONFIG.flatfield_dir, f"{profile_id}{_EXT}") + + @staticmethod + def _bake_gain(reference_path: str) -> Optional[np.ndarray]: + """Decode a reference like a negative (no WB, linear) and compute its gain map.""" + if not reference_path or not os.path.exists(reference_path): + return None + try: + from negpy.services.rendering.preview_manager import PreviewManager + + reference, _, _ = PreviewManager().load_linear_preview(reference_path, use_camera_wb=False, full_resolution=False) + return compute_gain(reference) + except Exception: + logger.exception("Flat-field: failed to decode reference %s", reference_path) + return None + + @staticmethod + def _write(profile_id: str, gain: np.ndarray, *, name: str, k1: float, source: str) -> None: + os.makedirs(APP_CONFIG.flatfield_dir, exist_ok=True) + np.savez_compressed( + FlatFieldProfiles._path_for_id(profile_id), + gain=gain.astype(np.float32), + token=gain_token(gain), + name=name, + k1=float(k1), + source=source, + ) + + @staticmethod + def _read(profile_id: str, keys: Tuple[str, ...]) -> Optional[Dict[str, np.ndarray]]: + """Read only the named members of a profile file (a zip — untouched members are + never decompressed, so metadata reads skip the ~MB gain array), or None if absent.""" + path = FlatFieldProfiles._path_for_id(profile_id) + if not profile_id or not os.path.exists(path): + return None + try: + with np.load(path, allow_pickle=False) as data: + return {k: data[k] for k in keys if k in data.files} + except Exception: + logger.exception("Flat-field: failed to read profile %s", profile_id) + return None + + @staticmethod + def create(name: str, reference_path: str, k1: float = 0.0) -> Optional[str]: + """Bake a reference into a new profile; returns its id (None if the decode failed).""" + gain = FlatFieldProfiles._bake_gain(reference_path) + if gain is None: + return None + return FlatFieldProfiles.import_gain(gain, name=name, k1=k1, source=reference_path) + + @staticmethod + def import_gain(gain: np.ndarray, *, name: str, k1: float = 0.0, source: str = "") -> str: + """Write an already-baked gain map as a new profile; returns its id.""" + profile_id = uuid.uuid4().hex + FlatFieldProfiles._write(profile_id, gain, name=name, k1=k1, source=source) + return profile_id + + @staticmethod + def load_gain(profile_id: str) -> Optional[Tuple[np.ndarray, str]]: + """(gain map, content token) for the render-path provider; None if missing/unreadable.""" + data = FlatFieldProfiles._read(profile_id, ("gain", "token")) + if data is None or "gain" not in data: + return None + gain = np.ascontiguousarray(data["gain"], dtype=np.float32) + token = str(data["token"]) if "token" in data else gain_token(gain) + return gain, token + + @staticmethod + def get(profile_id: str) -> Optional[FlatFieldProfile]: + """Profile metadata (name, k1, source), or None if the file is gone. + + Metadata-only read — the gain array is not decompressed, so this stays cheap + on the hot path (the sidebar rebuilds the profile list on every config sync). + """ + data = FlatFieldProfiles._read(profile_id, ("name", "k1", "source")) + if data is None: + return None + return FlatFieldProfile( + id=profile_id, + name=str(data["name"]) if "name" in data else profile_id, + k1=float(data["k1"]) if "k1" in data else 0.0, + source=str(data["source"]) if "source" in data else "", + ) + + @staticmethod + def list_profiles() -> List[Tuple[str, str]]: + """[(id, display name)] for existing profiles, sorted by name (case-insensitive).""" + directory = APP_CONFIG.flatfield_dir + if not os.path.isdir(directory): + return [] + out: List[Tuple[str, str]] = [] + for fname in os.listdir(directory): + if not fname.endswith(_EXT): + continue + prof = FlatFieldProfiles.get(fname[: -len(_EXT)]) + if prof is not None: + out.append((prof.id, prof.name)) + return sorted(out, key=lambda t: t[1].lower()) + + @staticmethod + def set_k1(profile_id: str, k1: float) -> None: + """Rewrite a profile's rig distortion in place, preserving its baked gain.""" + data = FlatFieldProfiles._read(profile_id, ("gain", "name", "source")) + if data is None or "gain" not in data: + return + FlatFieldProfiles._write( + profile_id, + np.ascontiguousarray(data["gain"], dtype=np.float32), + name=str(data["name"]) if "name" in data else profile_id, + k1=k1, + source=str(data["source"]) if "source" in data else "", + ) + + @staticmethod + def delete(profile_id: str) -> None: + try: + os.remove(FlatFieldProfiles._path_for_id(profile_id)) + except OSError: + pass diff --git a/negpy/services/assets/flatfield_migration.py b/negpy/services/assets/flatfield_migration.py new file mode 100644 index 00000000..7f924f49 --- /dev/null +++ b/negpy/services/assets/flatfield_migration.py @@ -0,0 +1,82 @@ +"""One-time migration of legacy DB-backed flat-field profiles to the npz file store. + +Before the file store, profiles lived in a ``flatfield_profiles`` table (name, path, +k1) and each per-image edit stored the resolved reference *path*. This bakes every +legacy profile's gain from its reference (if the file is still present), rewrites the +edits to reference the new profile by opaque id, remaps the active-profile setting, +then drops the old table. Best-effort and idempotent — guarded by a done flag, and it +never raises into app startup. +""" + +import json +import sqlite3 +from typing import Dict + +from negpy.kernel.system.logging import get_logger +from negpy.services.assets.flatfield import FlatFieldProfiles + +logger = get_logger(__name__) + +_DONE_FLAG = "flatfield_migrated_v2" + + +def _table_exists(conn: sqlite3.Connection, name: str) -> bool: + row = conn.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (name,)).fetchone() + return row is not None + + +def _rewrite_configs(conn: sqlite3.Connection, table: str, key_col: str, path_to_id: Dict[str, str]) -> None: + """Swap each edit's legacy ``reference_path`` for the new ``profile_id``.""" + rows = conn.execute(f"SELECT {key_col}, settings_json FROM {table}").fetchall() + for key, settings_json in rows: + if not settings_json or "reference_path" not in settings_json: + continue + try: + data = json.loads(settings_json) + except (ValueError, TypeError): + continue + if "reference_path" not in data: + continue + path = data.pop("reference_path") or "" + data["profile_id"] = path_to_id.get(path, "") + conn.execute( + f"UPDATE {table} SET settings_json = ? WHERE {key_col} = ?", + (json.dumps(data, default=str), key), + ) + + +def migrate_legacy_flatfield_profiles(repo) -> None: + """Bake legacy DB profiles into the npz store and repoint edits/settings at them.""" + if repo.get_global_setting(_DONE_FLAG): + return + try: + with sqlite3.connect(repo.edits_db_path) as conn: + if _table_exists(conn, "flatfield_profiles"): + legacy = conn.execute("SELECT name, path, k1 FROM flatfield_profiles").fetchall() + + name_to_id: Dict[str, str] = {} + path_to_id: Dict[str, str] = {} + for name, path, k1 in legacy: + new_id = FlatFieldProfiles.create(str(name), str(path or ""), float(k1 or 0.0)) + if new_id is None: + # Reference file gone — nothing to bake; the correction was already + # broken. Leave name/path unmapped so edits fall back to inactive. + logger.warning("Flat-field migration: could not bake profile %r (reference missing)", name) + continue + name_to_id[str(name)] = new_id + if path: + path_to_id[str(path)] = new_id + + _rewrite_configs(conn, "file_settings", "file_hash", path_to_id) + if _table_exists(conn, "edit_history"): + _rewrite_configs(conn, "edit_history", "rowid", path_to_id) + + conn.execute("DROP TABLE flatfield_profiles") + + active_name = repo.get_global_setting("flatfield_active_profile") + if active_name: + repo.save_global_setting("flatfield_active_profile", name_to_id.get(str(active_name), "")) + except Exception: + logger.exception("Flat-field migration failed; continuing without it") + + repo.save_global_setting(_DONE_FLAG, True) diff --git a/negpy/services/rendering/preview_manager.py b/negpy/services/rendering/preview_manager.py index 0bbd8da4..837593d4 100644 --- a/negpy/services/rendering/preview_manager.py +++ b/negpy/services/rendering/preview_manager.py @@ -398,7 +398,7 @@ def load_linear_preview_stitch( use_camera_wb: bool = False, full_resolution: bool = False, file_hash: str | None = None, - flatfield_path: str = "", + flatfield_profile_id: str = "", ) -> Tuple[ImageBuffer, Dimensions, dict]: """Assemble a stitch composite at preview scale by replaying the stored registration. Flat-field is applied per part here (a composite canvas must @@ -407,7 +407,7 @@ def load_linear_preview_stitch( Returned dims are the full-resolution canvas, matching the single-file convention of (original height, width) alongside a downsampled buffer. """ - flatfield = FlatFieldConfig(apply=bool(flatfield_path), reference_path=flatfield_path) + flatfield = FlatFieldConfig(apply=bool(flatfield_profile_id), profile_id=flatfield_profile_id) key = None if file_hash and color_space is not None: token = stitch_token(stitch) diff --git a/tests/test_batch_autocrop_worker.py b/tests/test_batch_autocrop_worker.py index c759bbd3..c5c13929 100644 --- a/tests/test_batch_autocrop_worker.py +++ b/tests/test_batch_autocrop_worker.py @@ -151,7 +151,7 @@ def test_batch_autocrop_applies_flatfield_and_crop_free_geometry( flatfield = replace( base.flatfield, apply=flatfield_enabled, - reference_path="/flat.dng", + profile_id="rig-1", k1=0.23, ) config = replace(base, geometry=geometry, flatfield=flatfield) diff --git a/tests/test_batch_progress.py b/tests/test_batch_progress.py index d691b471..2fd11e3a 100644 --- a/tests/test_batch_progress.py +++ b/tests/test_batch_progress.py @@ -107,7 +107,7 @@ def _task(name: str) -> ExportTask: a, b = _task("a.cr2"), _task("b.cr2") assert _same_decode_source(a, _task("a.cr2")) assert not _same_decode_source(a, b) - flat = replace(DEFAULT_WORKSPACE_CONFIG, flatfield=replace(DEFAULT_WORKSPACE_CONFIG.flatfield, apply=True, reference_path="/f.dng")) + flat = replace(DEFAULT_WORKSPACE_CONFIG, flatfield=replace(DEFAULT_WORKSPACE_CONFIG.flatfield, apply=True, profile_id="rig-1")) a_flat = ExportTask(file_info=a.file_info, params=flat, export_settings=preset) assert not _same_decode_source(a, a_flat) diff --git a/tests/test_config_deserialization.py b/tests/test_config_deserialization.py index e22642c7..54c48ee2 100644 --- a/tests/test_config_deserialization.py +++ b/tests/test_config_deserialization.py @@ -163,7 +163,7 @@ def test_explicit_mode_wins_over_legacy_use_original_res(self): def test_flatfield_apply_does_not_collide_with_rgbscan_enabled(self): """flatfield.apply and rgbscan.enabled must round-trip independently (#356).""" cfg = WorkspaceConfig( - flatfield=replace(WorkspaceConfig().flatfield, apply=True, reference_path="/r.dng"), + flatfield=replace(WorkspaceConfig().flatfield, apply=True, profile_id="abc123"), rgbscan=replace(WorkspaceConfig().rgbscan, enabled=False), ) back = WorkspaceConfig.from_flat_dict(cfg.to_dict()) diff --git a/tests/test_database_management.py b/tests/test_database_management.py index 74948f79..a2ff5335 100644 --- a/tests/test_database_management.py +++ b/tests/test_database_management.py @@ -19,7 +19,6 @@ def _seed(repo): repo.save_history_step("hash-a", 1, replace(cfg)) repo.save_file_mark("hash-a", "keeper") repo.save_normalization_roll("roll-1", (0.1, 0.1, 0.1), (0.9, 0.9, 0.9)) - repo.save_flatfield_profile("rig-1", "/refs/flat.dng", k1=-0.05) repo.save_export_presets([ExportPreset(name="mine")]) repo.save_global_setting("window_geometry", [0, 0, 800, 600]) repo.save_global_setting("last_open_folder", "/photos") @@ -34,7 +33,6 @@ def test_database_stats_counts_each_category(tmp_path): assert stats["edit_history"] == 2 assert stats["file_marks"] == 1 assert stats["normalization_rolls"] == 1 - assert stats["flatfield_profiles"] == 1 assert stats["export_presets"] == 1 # export_presets is one row inside global_settings; it must not inflate the # preferences count, which here is the two real settings we saved. @@ -51,7 +49,6 @@ def test_stats_on_empty_db_are_all_zero(tmp_path): "edit_history", "file_marks", "normalization_rolls", - "flatfield_profiles", "export_presets", "app_preferences", ): @@ -71,7 +68,6 @@ def test_clear_saved_edits_drops_only_per_image_data(tmp_path): assert stats["file_marks"] == 0 # Tooling kept. assert stats["normalization_rolls"] == 1 - assert stats["flatfield_profiles"] == 1 assert stats["export_presets"] == 1 assert stats["app_preferences"] == 2 @@ -103,7 +99,6 @@ def test_reset_everything_wipes_both_databases(tmp_path): "edit_history", "file_marks", "normalization_rolls", - "flatfield_profiles", "export_presets", "app_preferences", ): diff --git a/tests/test_flatfield_logic.py b/tests/test_flatfield_logic.py index 0f6acc88..3edd0b52 100644 --- a/tests/test_flatfield_logic.py +++ b/tests/test_flatfield_logic.py @@ -1,6 +1,5 @@ -import os - import numpy as np +import pytest from negpy.features.flatfield import logic as ff from negpy.features.flatfield.models import FlatFieldConfig @@ -15,34 +14,49 @@ def _radial_falloff(h: int, w: int) -> np.ndarray: return np.repeat(falloff[:, :, None], 3, axis=2).astype(np.float32) +@pytest.fixture +def gain_store(): + """Back the provider with an in-memory {id: (gain, token)} map; restore on exit.""" + store: dict[str, tuple[np.ndarray, str]] = {} + + def register(profile_id: str, reference: np.ndarray) -> None: + gain = ff.compute_gain(reference) + store[profile_id] = (gain, ff.gain_token(gain)) + + ff.set_gain_provider(lambda pid: store.get(pid)) + try: + yield register + finally: + ff.set_gain_provider(None) + + def test_disabled_is_noop(): img = np.full((16, 16, 3), 0.5, dtype=np.float32) - out = ff.apply_flatfield(img, FlatFieldConfig(apply=False, reference_path="/nope.dng")) + out = ff.apply_flatfield(img, FlatFieldConfig(apply=False, profile_id="anything")) + assert out is img + + +def test_empty_profile_is_noop(): + img = np.full((16, 16, 3), 0.5, dtype=np.float32) + out = ff.apply_flatfield(img, FlatFieldConfig(apply=True, profile_id="")) assert out is img -def test_empty_path_is_noop(): +def test_unknown_profile_is_noop(gain_store): img = np.full((16, 16, 3), 0.5, dtype=np.float32) - out = ff.apply_flatfield(img, FlatFieldConfig(apply=True, reference_path="")) + out = ff.apply_flatfield(img, FlatFieldConfig(apply=True, profile_id="ghost")) assert out is img + assert ff.flatfield_token(FlatFieldConfig(apply=True, profile_id="ghost")) == "" -def test_correction_flattens_uneven_illumination(tmp_path): +def test_correction_flattens_uneven_illumination(gain_store): h, w = 128, 192 falloff = _radial_falloff(h, w) + gain_store("rig", falloff) # A uniform scene captured under this illumination is just the falloff map. captured = falloff.copy() - - # Seed the gain cache as if `falloff` had been decoded from a reference file, - # so apply_flatfield runs end-to-end without a real RAW decode. - ref_file = tmp_path / "ref.dng" - ref_file.write_bytes(b"x") - path = str(ref_file) - ff._GAIN_CACHE[(path, os.path.getmtime(path))] = ff._compute_gain(falloff) - - cfg = FlatFieldConfig(apply=True, reference_path=path) - corrected = ff.apply_flatfield(captured, cfg) + corrected = ff.apply_flatfield(captured, FlatFieldConfig(apply=True, profile_id="rig")) # Before: clearly uneven. After: near-flat across the field. assert captured.std() > 0.05 @@ -50,14 +64,31 @@ def test_correction_flattens_uneven_illumination(tmp_path): assert corrected.dtype == np.float32 -def test_gain_resized_to_image(tmp_path): - # Gain computed at one size must resize to a differently-sized working image. - falloff = _radial_falloff(64, 64) - ref_file = tmp_path / "ref2.dng" - ref_file.write_bytes(b"x") - path = str(ref_file) - ff._GAIN_CACHE[(path, os.path.getmtime(path))] = ff._compute_gain(falloff) - +def test_gain_resized_to_image(gain_store): + # Gain baked at one size must resize to a differently-sized working image. + gain_store("rig", _radial_falloff(64, 64)) img = np.full((100, 140, 3), 0.5, dtype=np.float32) - out = ff.apply_flatfield(img, FlatFieldConfig(apply=True, reference_path=path)) + out = ff.apply_flatfield(img, FlatFieldConfig(apply=True, profile_id="rig")) assert out.shape == img.shape + + +def test_token_is_stable_and_profile_scoped(gain_store): + gain_store("a", _radial_falloff(64, 64)) + gain_store("b", _radial_falloff(48, 72)) + cfg_a = FlatFieldConfig(apply=True, profile_id="a") + + tok = ff.flatfield_token(cfg_a) + assert tok.startswith("|ff:a:") + assert tok == ff.flatfield_token(cfg_a) # deterministic + assert tok != ff.flatfield_token(FlatFieldConfig(apply=True, profile_id="b")) + assert ff.flatfield_token(FlatFieldConfig(apply=False, profile_id="a")) == "" + + +def test_invalidate_drops_cache(gain_store): + gain_store("rig", _radial_falloff(64, 64)) + cfg = FlatFieldConfig(apply=True, profile_id="rig") + assert ff.flatfield_token(cfg) != "" # populates the cache + + ff.set_gain_provider(lambda pid: None) # provider now yields nothing + ff.invalidate_gain("rig") + assert ff.flatfield_token(cfg) == "" diff --git a/tests/test_flatfield_migration.py b/tests/test_flatfield_migration.py new file mode 100644 index 00000000..ea6c3454 --- /dev/null +++ b/tests/test_flatfield_migration.py @@ -0,0 +1,81 @@ +import json +import sqlite3 + +import numpy as np +import pytest + +from negpy.infrastructure.storage.repository import StorageRepository +from negpy.services.assets import flatfield as ffstore +from negpy.services.assets.flatfield import FlatFieldProfiles +from negpy.services.assets.flatfield_migration import migrate_legacy_flatfield_profiles + + +@pytest.fixture +def legacy_repo(tmp_path, monkeypatch): + """A repo carrying a legacy flatfield_profiles table + a per-image edit pointing at one.""" + monkeypatch.setattr(ffstore.APP_CONFIG, "flatfield_dir", str(tmp_path / "flatfield"), raising=False) + monkeypatch.setattr(FlatFieldProfiles, "_bake_gain", staticmethod(lambda path: np.ones((8, 8, 3), dtype=np.float32))) + + repo = StorageRepository(str(tmp_path / "edits.db"), str(tmp_path / "settings.db")) + repo.initialize() + + with sqlite3.connect(repo.edits_db_path) as conn: + conn.execute("CREATE TABLE flatfield_profiles (name TEXT PRIMARY KEY, path TEXT, k1 REAL DEFAULT 0.0)") + conn.executemany( + "INSERT INTO flatfield_profiles (name, path, k1) VALUES (?, ?, ?)", + [("rig-a", str(tmp_path / "a.dng"), -0.05), ("rig-b", str(tmp_path / "b.dng"), 0.0)], + ) + edit = {"apply": True, "reference_path": str(tmp_path / "a.dng"), "k1": -0.05, "density": 1.0} + conn.execute( + "INSERT INTO file_settings (file_hash, settings_json) VALUES (?, ?)", + ("hash1", json.dumps(edit)), + ) + # References were baked from real files; touch them so create() sees them present. + (tmp_path / "a.dng").write_bytes(b"x") + (tmp_path / "b.dng").write_bytes(b"x") + repo.save_global_setting("flatfield_active_profile", "rig-a") + return repo + + +def _config(repo, file_hash): + with sqlite3.connect(repo.edits_db_path) as conn: + row = conn.execute("SELECT settings_json FROM file_settings WHERE file_hash = ?", (file_hash,)).fetchone() + return json.loads(row[0]) + + +def test_migration_bakes_and_repoints(legacy_repo): + migrate_legacy_flatfield_profiles(legacy_repo) + + # Both legacy profiles are now npz files, keyed by opaque id. + profiles = dict((name, pid) for pid, name in FlatFieldProfiles.list_profiles()) + assert set(profiles) == {"rig-a", "rig-b"} + assert FlatFieldProfiles.get(profiles["rig-a"]).k1 == -0.05 + + # The per-image edit now references rig-a by id, with the legacy path dropped. + cfg = _config(legacy_repo, "hash1") + assert cfg["profile_id"] == profiles["rig-a"] + assert "reference_path" not in cfg + + # Active-profile setting remapped name -> id; legacy table gone; flag set. + assert legacy_repo.get_global_setting("flatfield_active_profile") == profiles["rig-a"] + with sqlite3.connect(legacy_repo.edits_db_path) as conn: + assert conn.execute("SELECT name FROM sqlite_master WHERE name='flatfield_profiles'").fetchone() is None + assert legacy_repo.get_global_setting("flatfield_migrated_v2") is True + + +def test_migration_is_idempotent(legacy_repo): + migrate_legacy_flatfield_profiles(legacy_repo) + count_after_first = len(FlatFieldProfiles.list_profiles()) + + migrate_legacy_flatfield_profiles(legacy_repo) # second run must not re-bake + assert len(FlatFieldProfiles.list_profiles()) == count_after_first + + +def test_migration_no_legacy_table_just_sets_flag(tmp_path, monkeypatch): + monkeypatch.setattr(ffstore.APP_CONFIG, "flatfield_dir", str(tmp_path / "flatfield"), raising=False) + repo = StorageRepository(str(tmp_path / "edits.db"), str(tmp_path / "settings.db")) + repo.initialize() + + migrate_legacy_flatfield_profiles(repo) + assert repo.get_global_setting("flatfield_migrated_v2") is True + assert FlatFieldProfiles.list_profiles() == [] diff --git a/tests/test_flatfield_profiles.py b/tests/test_flatfield_profiles.py new file mode 100644 index 00000000..1602eb30 --- /dev/null +++ b/tests/test_flatfield_profiles.py @@ -0,0 +1,100 @@ +import os + +import numpy as np +import pytest + +from negpy.features.flatfield import logic as ff +from negpy.features.flatfield.models import FlatFieldConfig +from negpy.services.assets import flatfield as ffstore +from negpy.services.assets.flatfield import FlatFieldProfiles + + +def _gain(h=24, w=32) -> np.ndarray: + yy, xx = np.mgrid[0:h, 0:w].astype(np.float32) + g = 1.0 + 0.5 * (xx / w) + 0.25 * (yy / h) + return np.repeat(g[:, :, None], 3, axis=2).astype(np.float32) + + +@pytest.fixture +def store_dir(tmp_path, monkeypatch): + """Point the profile store at a temp dir and stub the RAW decode/bake.""" + monkeypatch.setattr(ffstore.APP_CONFIG, "flatfield_dir", str(tmp_path / "flatfield"), raising=False) + monkeypatch.setattr(FlatFieldProfiles, "_bake_gain", staticmethod(lambda path: _gain())) + return tmp_path + + +def test_import_and_load_round_trip(store_dir): + gain = _gain() + pid = FlatFieldProfiles.import_gain(gain, name="Rig A", k1=-0.08, source="/refs/flat.dng") + + loaded, token = FlatFieldProfiles.load_gain(pid) + assert np.array_equal(loaded, gain) + assert token == ff.gain_token(gain) + + prof = FlatFieldProfiles.get(pid) + assert (prof.id, prof.name, prof.k1, prof.source) == (pid, "Rig A", -0.08, "/refs/flat.dng") + + +def test_opaque_ids_are_unique_per_profile(store_dir): + a = FlatFieldProfiles.import_gain(_gain(), name="dup") + b = FlatFieldProfiles.import_gain(_gain(), name="dup") + assert a != b # same display name, distinct ids + assert {n for _, n in FlatFieldProfiles.list_profiles()} == {"dup"} + assert sorted(i for i, _ in FlatFieldProfiles.list_profiles()) == sorted([a, b]) + + +def test_list_profiles_sorted_by_name(store_dir): + FlatFieldProfiles.import_gain(_gain(), name="Zeta") + FlatFieldProfiles.import_gain(_gain(), name="alpha") + assert [n for _, n in FlatFieldProfiles.list_profiles()] == ["alpha", "Zeta"] + + +def test_set_k1_preserves_gain_and_token(store_dir): + pid = FlatFieldProfiles.import_gain(_gain(), name="rig", k1=0.0) + _, tok_before = FlatFieldProfiles.load_gain(pid) + + FlatFieldProfiles.set_k1(pid, 0.12) + gain_after, tok_after = FlatFieldProfiles.load_gain(pid) + assert FlatFieldProfiles.get(pid).k1 == 0.12 + assert np.array_equal(gain_after, _gain()) # gain untouched by a k1 edit + assert tok_after == tok_before # so the render cache is not invalidated + + +def test_delete_and_missing(store_dir): + pid = FlatFieldProfiles.import_gain(_gain(), name="rig") + FlatFieldProfiles.delete(pid) + assert FlatFieldProfiles.load_gain(pid) is None + assert FlatFieldProfiles.get(pid) is None + FlatFieldProfiles.delete(pid) # idempotent, no raise + + +def test_create_bakes_and_activates_via_provider(store_dir): + pid = FlatFieldProfiles.create("rig", "/refs/flat.dng") + assert pid is not None + + ff.set_gain_provider(FlatFieldProfiles.load_gain) + try: + stored, _ = FlatFieldProfiles.load_gain(pid) + img = np.full((24, 32, 3), 0.5, dtype=np.float32) + out = ff.apply_flatfield(img, FlatFieldConfig(apply=True, profile_id=pid)) + assert np.allclose(out, img * stored) # provider resolved the baked gain and applied it + finally: + ff.set_gain_provider(None) + + +def test_create_returns_none_when_reference_unreadable(store_dir, monkeypatch): + monkeypatch.setattr(FlatFieldProfiles, "_bake_gain", staticmethod(lambda path: None)) + assert FlatFieldProfiles.create("rig", "/gone.dng") is None + + +def test_metadata_reads_do_not_require_the_gain(store_dir): + # A file with metadata but no gain member: get()/list_profiles must still work, + # proving the sidebar's per-sync list build never decompresses the gain array. + directory = ffstore.APP_CONFIG.flatfield_dir + os.makedirs(directory, exist_ok=True) + np.savez_compressed(os.path.join(directory, "meta-only.npz"), name="MetaOnly", k1=0.05, source="/x.dng") + + prof = FlatFieldProfiles.get("meta-only") + assert prof is not None and prof.name == "MetaOnly" and prof.k1 == 0.05 + assert ("meta-only", "MetaOnly") in FlatFieldProfiles.list_profiles() + assert FlatFieldProfiles.load_gain("meta-only") is None # no gain member to resolve diff --git a/tests/test_override_config.py b/tests/test_override_config.py index 0e9f205a..130ed365 100644 --- a/tests/test_override_config.py +++ b/tests/test_override_config.py @@ -30,6 +30,7 @@ def _make_app_config(**kwargs) -> AppConfig: user_icc_dir="/tmp/icc", crosstalk_dir="/tmp/crosstalk", sensor_dir="/tmp/sensor", + flatfield_dir="/tmp/flatfield", gear_dir="/tmp/gear", contact_sheet_templates_dir="/tmp/contact_sheets", default_export_dir="/tmp/export", diff --git a/tests/test_parallel_dispatch.py b/tests/test_parallel_dispatch.py index d3cf8db2..42385930 100644 --- a/tests/test_parallel_dispatch.py +++ b/tests/test_parallel_dispatch.py @@ -128,6 +128,7 @@ def test_override_parsing_and_apply(): user_icc_dir="", crosstalk_dir="", sensor_dir="", + flatfield_dir="", gear_dir="", contact_sheet_templates_dir="", default_export_dir="", diff --git a/tests/test_flatfield_profile_repo.py b/tests/test_storage_repo.py similarity index 55% rename from tests/test_flatfield_profile_repo.py rename to tests/test_storage_repo.py index e095a0bb..67bf37c2 100644 --- a/tests/test_flatfield_profile_repo.py +++ b/tests/test_storage_repo.py @@ -1,4 +1,3 @@ -import os import sqlite3 from negpy.infrastructure.storage.repository import StorageRepository @@ -10,38 +9,6 @@ def _repo(tmp_path): return repo -def test_profile_k1_round_trip(tmp_path): - repo = _repo(tmp_path) - repo.save_flatfield_profile("rig-a", "/refs/flat.dng", k1=-0.08) - assert repo.get_flatfield_profile("rig-a") == ("/refs/flat.dng", -0.08) - - # Default k1 when omitted. - repo.save_flatfield_profile("rig-b", "/refs/b.dng") - assert repo.get_flatfield_profile("rig-b") == ("/refs/b.dng", 0.0) - - assert repo.get_flatfield_profile("missing") is None - - -def test_k1_column_migration_on_legacy_db(tmp_path): - """A DB created before the k1 column must gain it (defaulting to 0.0) on init.""" - edits = str(tmp_path / "edits.db") - conn = sqlite3.connect(edits) - try: - conn.execute("CREATE TABLE flatfield_profiles (name TEXT PRIMARY KEY, path TEXT)") - conn.execute("INSERT INTO flatfield_profiles (name, path) VALUES (?, ?)", ("legacy", "/refs/old.dng")) - conn.commit() - finally: - conn.close() - - repo = StorageRepository(edits, str(tmp_path / "settings.db")) - repo.initialize() # runs the ALTER TABLE ADD COLUMN migration - - assert repo.get_flatfield_profile("legacy") == ("/refs/old.dng", 0.0) - repo.save_flatfield_profile("legacy", "/refs/old.dng", k1=0.15) - assert repo.get_flatfield_profile("legacy") == ("/refs/old.dng", 0.15) - assert os.path.exists(edits) - - def test_save_global_settings_batch_round_trip(tmp_path): repo = _repo(tmp_path) values = {"a": 1, "b": [1, 2], "c": {"x": "y"}, "d": True, "e": "text"}