Skip to content

API.md

Latest commit

 

History

History
234 lines (187 loc) · 8.78 KB

API.md

File metadata and controls

234 lines (187 loc) · 8.78 KB
Raw

SHARP Image2Splat — API Reference

HTTP service that turns an RGB image (+ optionally a metric depth EXR) into a 3D Gaussian Splat (.ply). Single-frame, synchronous, no auth.

  • Protocol: HTTP/1.1, multipart/form-data for uploads.
  • Default port: 8765.
  • Base URL: http://<host>:8765
    • On the GPU host itself: http://localhost:8765
    • From another machine: http://<gpu-host>:8765 (LAN / VPN address of the GPU host)

The server holds one SHARP model in GPU memory and processes requests serially. A /generate call typically takes ~4–8 s (longer for exr_grade, which runs two inference passes). Set client timeouts to ≥ 180 s.

Endpoints

Method Path Purpose
GET /health liveness / readiness
POST /generate RGB (+ depth) → splat .ply
POST /inspect report stats for a depth EXR
GET /docs interactive Swagger UI (browser)

GET /health

No parameters. Returns JSON:

{ "ok": true, "cuda": true, "device": "cuda", "model_loaded": true }
  • model_loaded: false → the checkpoint is still loading at startup; wait and retry.
  • cuda: false → the server is on CPU (misconfigured); generation will be very slow.

POST /generate

multipart/form-data. Produces a Gaussian splat .ply.

Fields

field kind required default notes
image file yes — RGB PNG/JPG. Alpha is stripped. Any resolution.
depth file conditional — Single-channel float32 EXR, camera-space Z in metres, positive forward. Required for exr_pixel and exr_grade; ignored for sharp. Sky / no-hit pixels (0 or non-finite) are remapped to a far value automatically.
albedo file conditional — Albedo-AOV PNG/JPG. Required only for exr_grade with grade_source=region.
f_px form float conditional — Focal length in pixels. Provide this or both focal_mm + aperture_mm.
focal_mm form float conditional — Focal length in mm (with aperture_mm).
aperture_mm form float conditional — Horizontal sensor aperture in mm. f_px = focal_mm / aperture_mm * image_width.
depth_method form string no exr_pixel One of sharp, exr_pixel, exr_grade.
grade_curve form string no affine exr_grade only: affine, polynomial, histogram.
grade_source form string no percentile exr_grade only: percentile, region.
grade_min_slope form float no 0.0 exr_grade only. Floor on the grade curve's slope to stop depth compression / 3DGS flicker ("popping") under camera motion. 0 = off; 0.3–0.6 reduces popping at the cost of a looser metric match. Must be >= 0.
blend_alpha form float no 0.4 exr_pixel only. 0 = trust the depth fully, 1 = vanilla SHARP. Ignored by sharp; internally forced to 0 by exr_grade.
output_path form string no — If set, the server writes the .ply to this server-side path and returns JSON instead of the bytes. Use only when the caller shares a filesystem with the GPU host.

You must always supply focal information: either f_px, or both focal_mm and aperture_mm.

depth_method — what each one does

  • sharp — Plain SHARP. No depth injection. The splat uses SHARP's own predicted (monocular) depth at its native metric scale. depth is not needed.
  • exr_pixel — Per-pixel inverse-depth blend of the EXR into SHARP's prediction, controlled by blend_alpha. Anchors metric scale per pixel.
  • exr_grade — Globally remaps SHARP's own predicted depth so its value distribution matches the EXR's (a "colour grade" of the depth). Value-based, not per-pixel, so it is robust to small misalignment between the RGB and the EXR.
    • grade_source=percentile — match depth percentiles of the two distributions (no albedo needed).
    • grade_source=region — match per-region median depths; regions are segmented from the albedo AOV (requires the albedo upload).
    • grade_curve — how the mapping is fit: affine (scale+shift), polynomial, or histogram (a monotonic lookup curve; richest).
    • grade_min_slope — if the splat flickers under camera motion (3DGS depth-sort "popping"), the grade curve is compressing depth somewhere. Set this to 0.3–0.6 to floor the curve's slope so depth can't collapse into coplanar bands. 0 = off.

Required inputs by method

depth_method grade_source needs image needs depth needs albedo
sharp — ✅ — —
exr_pixel — ✅ ✅ —
exr_grade percentile ✅ ✅ —
exr_grade region ✅ ✅ ✅

Response

Default — the raw .ply bytes:

  • Content-Type: application/octet-stream
  • Content-Disposition: attachment; filename="splat.ply"
  • Headers:
    • X-Num-Gaussians — number of Gaussians in the splat
    • X-F-Px — the focal length (px) used
    • X-Depth-Method — the method used (echoes the request)

If output_path was provided — JSON, no bytes:

{
  "ok": true,
  "path": "<output_path>",
  "num_gaussians": 1179648,
  "f_px": 877.71,
  "image_size": [1080, 1920],
  "depth_method": "exr_grade"
}

Status codes

code meaning
200 success
422 invalid/missing parameters: unknown depth_method/grade_curve/grade_source; depth missing for exr_pixel/exr_grade; albedo missing for exr_grade+region; no focal info; bad argument combination
500 prediction failed (unexpected server error); body has {"detail": "..."}
503 model not loaded yet — retry shortly

Error bodies are JSON: {"detail": "<message>"}.

POST /inspect

multipart/form-data with a single depth file. Use to sanity-check a depth EXR before generating. Returns JSON:

{
  "ok": true,
  "shape": [1080, 1920],
  "dtype": "float32",
  "min": 12.57,
  "max": 9988.56,
  "mean": 297.69,
  "sky_or_far_pct": 26.2,
  "all_finite": true
}

Examples

curl (the ^ line-continuation is for Windows cmd; use \ on macOS/Linux):

Plain SHARP (no depth):

curl http://<host>:8765/generate ^
  -F image=@beauty.png ^
  -F focal_mm=24 -F aperture_mm=35 ^
  -F depth_method=sharp ^
  -o splat.ply

Per-pixel EXR injection:

curl http://<host>:8765/generate ^
  -F image=@beauty.png -F depth=@depth.exr ^
  -F focal_mm=24 -F aperture_mm=35 ^
  -F depth_method=exr_pixel -F blend_alpha=0.4 ^
  -o splat.ply

Global grade, percentile + histogram:

curl http://<host>:8765/generate ^
  -F image=@beauty.png -F depth=@depth.exr ^
  -F f_px=877.71 ^
  -F depth_method=exr_grade -F grade_source=percentile -F grade_curve=histogram ^
  -o splat.ply

Global grade, region (needs albedo):

curl http://<host>:8765/generate ^
  -F image=@beauty.png -F depth=@depth.exr -F albedo=@albedo.png ^
  -F focal_mm=24 -F aperture_mm=35 ^
  -F depth_method=exr_grade -F grade_source=region -F grade_curve=affine ^
  -o splat.ply

Inspect a depth EXR:

curl http://<host>:8765/inspect -F depth=@depth.exr

Python

import requests

URL = "http://<host>:8765/generate"

files = {
    "image":  open("beauty.png", "rb"),
    "depth":  open("depth.exr", "rb"),      # omit for depth_method="sharp"
    "albedo": open("albedo.png", "rb"),     # only for grade_source="region"
}
data = {
    "focal_mm": 24, "aperture_mm": 35,      # or: "f_px": 877.71
    "depth_method": "exr_grade",
    "grade_source": "region",
    "grade_curve": "histogram",
}

r = requests.post(URL, files=files, data=data, timeout=180)
r.raise_for_status()
with open("splat.ply", "wb") as f:
    f.write(r.content)
print(r.headers["X-Num-Gaussians"], "gaussians,", r.headers["X-Depth-Method"])

Notes

  • Coordinate convention: OpenCV (x right, y down, z forward); the splat's scene centre is roughly at (0, 0, +z). The output .ply colours are sRGB (compatible with standard 3DGS viewers, e.g. SuperSplat / PlayCanvas).
  • Depth must be camera-space Z, in metres, positive forward — not ray distance and not world-space Z. If your scene is not in metres, convert before exporting.
  • f_px and image resolution: f_px is in the uploaded image's pixel frame. If an upstream model upscaled the image, compute f_px against the uploaded width (f_px = focal_mm/aperture_mm * uploaded_width).
  • PowerShell: native multipart (Invoke-WebRequest -Form) requires PowerShell 7+. On Windows PowerShell 5.1 use curl.exe or the Python example.
  • Output size: a splat .ply is roughly 60 MB at the default Gaussian count. Plan transfer/storage accordingly, or pass output_path to write it server-side.