Skip to content

README.md

Latest commit

 

History

History
97 lines (68 loc) · 5.3 KB

README.md

File metadata and controls

97 lines (68 loc) · 5.3 KB
Raw

Sharp Depth Injection

External-depth-prior extension to Apple's ml-sharp Gaussian Splat predictor. Feeds a metric depth map (e.g. a Houdini cam_zdepth AOV, or any camera-space Z depth) into SHARP at inference time so the resulting splat has the correct metric scale and plausible parallax under modest camera moves.

Typical pipeline it was built for:

  1. 3D scene (Houdini / Blender / etc.) — beauty render + a camera-space depth AOV (Karma cam_zdepth / hitPz)
  2. Image model (an img2img pass, e.g. Nano Banana 2 / Gemini Flash Image) — a photoreal RGB from the beauty
  3. SHARP (this extension) — input = RGB + metric depth → metric Gaussian Splat

Use case: film / previz — the quality bar is that camera moves of a few degrees of orbit or a metre of dolly look parallactically plausible. The subclass + runtime hot-swap is the only modification path; Apple's ml-sharp is never edited.

Layout

The extension is its own package alongside an unmodified ml-sharp checkout (a sibling directory):

<parent>/
├── ml-sharp/                  ← apple/ml-sharp @ cdb4ddc6, untouched
└── Sharp_Depth_Injection/     ← this repo
    └── sharp_ext/
        ├── external_depth.py  ← ExternalDepthGaussianComposer subclass
        ├── swap_composer.py   ← runtime hot-swap into a built predictor
        └── predict_with_depth.py

Install

# 1. Clone Apple's ml-sharp at the pinned commit, as a sibling of this repo
git clone https://github.com/apple/ml-sharp.git ../ml-sharp
git -C ../ml-sharp checkout cdb4ddc6

# 2. Create a Python 3.13 venv inside this repo, and activate it
python -m venv .venv
#   Windows:  .venv\Scripts\Activate.ps1      macOS/Linux:  source .venv/bin/activate

# 3. Install ml-sharp's deps + ml-sharp itself + this extension
pip install -r ../ml-sharp/requirements.txt
pip install -e ../ml-sharp        # so `import sharp` resolves
pip install -e .                  # so `import sharp_ext` resolves

CUDA torch: ml-sharp's requirements pull a CPU-only torch by default (silently runs on CPU). Force the CUDA build for your CUDA version, e.g. pip install --force-reinstall torch torchvision --index-url https://download.pytorch.org/whl/cu128, then verify python -c "import torch; print(torch.cuda.is_available())" prints True.

The SHARP checkpoint downloads automatically on first inference to ~/.cache/torch/hub/checkpoints/.

Verification

Set SHARP_TEST_IMAGE to any local RGB (falls back to ../ml-sharp/data/teaser.jpg if present). From the activated venv at the repo root:

python tests/test_decoder_stride.py     # decoder grid resolution (stride 2 -> 768 grid)
python tests/test_self_consistency.py   # feed SHARP's own depth back, expect near-identity

Usage

Depth must be camera-space Z, in metres, positive forward (not ray distance, not world-Z). Sky / no-hit pixels that read 0 are remapped to a finite far distance by load_depth_exr, so SHARP doesn't plant fake near-camera Gaussians where the sky is.

import imageio.v3 as iio
from sharp_ext import load_depth_exr, predict_image_with_depth
from tests._common import build_predictor, pick_device

device = pick_device()
predictor = build_predictor(device)

image = iio.imread("beauty.png")          # H×W×3 uint8
depth = load_depth_exr("depth.exr")       # H×W float32, metres, +Z forward

# f_px = focal_mm / horizontal_aperture_mm * image_width
gaussians = predict_image_with_depth(
    predictor, image, f_px=877.71,
    external_depth=depth, blend_alpha=0.4,
)

See SINGLE_FRAME_INTEGRATION.md for a self-contained brief to embed the single-frame path in another app.

HTTP service

A FastAPI service wraps this library for image (+ metric depth EXR) → .ply over HTTP — see service/README.md and service/API.md. It runs on :8765 (one model held in VRAM) and exposes three depth_methods: sharp, exr_pixel (per-pixel blend), and exr_grade (remap SHARP's own predicted depth to the EXR's distribution). For splats prefer exr_grade (grade_source=region, grade_curve=polynomial, grade_min_slope=1.0): it scales SHARP's coherent geometry instead of injecting depth per pixel, which avoids the flying / smeared gaussians exr_pixel produces at silhouettes.

The ~3.4× NDC scale offset

The released checkpoint applies a fixed internal NDC scaling, so an injected metric depth D lands at world-Z ≈ 3.4·D (both injection methods, uniform). A metric_rescale + far_cap fix was tried and reverted (far_cap flattened far geometry onto a wall). Correct the scale downstream in your consumer with a single uniform factor — divide positions and Gaussian scales by the measured factor (~3.4) — rather than in the shared service. Plain sharp (no injection) is unaffected. See CLAUDE.md for the full note.

Status

Used in production: the extension + FastAPI service (service/serve.py) run as an auto-start service (:8765, one CUDA model held resident) driven over HTTP by downstream apps. Three depth methods ship (sharp / exr_pixel / exr_grade); the working splat recipe is exr_grade / region / polynomial / min_slope 1.0. Self-consistency + decoder-stride tests pass.