Dev/calibration#9
Open
romleiaj wants to merge 216 commits into
Open
Conversation
…urn it on with /sys/arch/use_nvjpeg.
…PU. Make SensorProfiles location relative, not hardcoded.
…upport GPU. Make SensorProfiles location relative, not hardcoded." This reverts commit 043f233. Move phase one SDK changes to separate branch.
Confirmed functionality on nuvo0, and improved write time from ~0.8s to ~0.2s.
…y are failing to write to disk. Remove hardcoded path, move into header file.
Successful build, no delta seen. Hopefully makes system more robust to stop/starts.
…e location needs modification for startup scripts.
Dev/startup refactor
…ne class. Delete old shims and consul work.
calibrate_rig.py now runs the entire flight calibration end to end. Per modality group it resolves a model by, in order: an existing aligned/ model (--reuse-aligned), an existing feature-extracted database, or building one from images0. After exporting all camera models it writes registration QC gifs. - build_colmap_database (rig.py): pycolmap-native SIFT extraction (one OPENCV camera per folder, KAMERA-tuned settings) + prior-based spatial matching, replacing the docker feature_extractor/matcher scripts and cutting matching from O(N^2) to spatial neighbors. - registration_gifs.py: flip each non-reference camera against its colocated reference-modality image warped into its view, driven purely by the exported StandardCamera models. Validated on fl09 end to end: 9 camera models + 6 gifs (uv/ir vs rgb at each station), all frames 2-up and correctly sized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
prepare_flight.py reorganizes a raw flight (per-view folders holding all modalities + meta jsons) into the calibration pipeline's layout: <flight>/colmap_<group>/images0/<prefix>_<channel>_<modality>/, split by modality group (rgb+uv vs ir) and symlinked by default. Synchronized triggers are kept together, as the rig/frame logic requires. Frame selection (flight_prep.select_by_spacing) pares by 3-D INS spacing, every-Nth, and/or a max count. It also estimates forward overlap from the along-track ground footprint and flags when a selection is too sparse for SfM at the lowest altitude -- the binding constraint. On TO26Su1 fl004 (3-camera EO calibration, three figure-8 altitude bands at ~275/435/565 m): the flight is not spatially oversampled -- at its native ~44 m trigger spacing forward overlap is already ~55%, and any spatial thinning drops the low band below the SfM floor. So the right selection there is keep-all; the compute win is spatial matching, not decimation. Staged all 2333 triggers x 3 cameras. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Alongside the per-camera v3 yamls, calibrate_rig.py now writes one self-contained <flight>_<date>_<config>_rig.json describing the whole mount: - provenance: flight / effort / config / date parsed from the imagery, UTC calibration time, method, pycolmap version, git commit; - reference frame: INS-body platform, ENU world with its lat0/lon0/h0 origin, and the quaternion convention spelled out; - per group (rgb, ir): reference camera, model source, boresight (ins_from_rig), lever arm, and boresight residual stats; - per camera: intrinsics (fx/fy/cx/cy + OpenCV distortion), the INS mount (camera_quaternion + position), the rig extrinsics (sensor_from_rig rotation + translation), median reprojection error, and image count. The reference sensor's sensor_from_rig is identity by construction; the file is the authoritative mount model, with the yamls as the per-camera runtime view. Verified on fl09 (EO+IR): fl09_20200830_85mm_25_5deg_rig.json, 2 groups, 9 cameras, valid and round-trippable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents the two-step rig calibration end to end using fl004 as the worked example: prepare_flight.py (stage raw per-view imagery into the images0 per-camera layout, with frame-selection guidance and the overlap caveat) then calibrate_rig.py (build database, prior-position map, boresight, export). Covers the raw folder layout, outputs (per-camera v3 yamls, the rig JSON, QC gifs), a one-paragraph how-it- works, and troubleshooting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirror the main README's native post-processing setup (conda env create + make install on Linux/macOS, pip install -e . on Windows), replacing the local micromamba-specific note. Keep the README's guidance that conda auto-selects the CUDA vs CPU pycolmap build and that the GPU only matters for this calibration step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fl004 as staged is RGB, but the pipeline handles rgb/uv/ir the same way, so stop framing the flight as inherently EO-only: raw folders hold whatever modalities were flown, the images0 tree shows both colmap_rgb (rgb+uv) and colmap_ir groups, each group runs only if its workspace exists, and outputs list one yaml per physical camera (nine for a 3-station EO+UV+IR flight). fl004 stays as the concrete example without implying single-modality is the norm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First-time setup should come before the quick-run commands. Dropped the now-redundant "see Installation below" pointer from the TL;DR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cut ~25% of the prose (1609 -> 1185 words) — condensed the install to a single block, merged duplicated commands, trimmed the flag/output sections — while keeping every caveat (GPU, INS-from-meta, don't over-prune, reprojection-is-health-signal). Added four image slots (reconstruction overview, fl004 trajectory, registration gif, mount frame diagram) with captions, and an assets/calibration/ manifest listing what each should show. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-existing doc polish on this branch: inline flowcharts for the staging/calibration pipeline, the calibrate_rig solve fan-out, and the frame chain (replacing the planned mount_frames.png), plus yaml/gif naming cleanup in the assets checklist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vismatch (image-matching-models) wraps MINIMA-LoFTR and auto-downloads its weights; it pulls in torch, so it is opt-in via `uv sync --group fusion`. kornia is pinned <0.8 because MINIMA's vendored LoFTR imports kornia.utils.grid, removed in 0.8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deep_match.py wraps vismatch/MINIMA-LoFTR behind a lazy import (the pipeline stays importable without torch) with a tested coordinate contract: matched keypoints come back in the frames of the arrays passed in. fusion.py registers each IR image into the EO reconstruction, visual-localization style: - eo_partners pairs each IR image with its co-located same-trigger EO image by (station, frame key) — never by filename swaps, whose extensions differ — with a temporal cap so IR triggers whose EO fell out of the model don't pair across banking turns. - warp_homography/prewarp_eo pre-align the EO image into the IR view via a ground-plane homography from the per-group mounts (exactly invertible; INTER_AREA pre-resize to avoid aliasing through the ~20x scale gap), so the matcher only faces the modality gap. On fl09 this yields 1000-3500 matches/pair at 2-6 px residual. - lift_eo_pixels casts each matched EO pixel from the image's reconstructed pose to the interpolated depth of its triangulated points — EO images carry only ~250-500 SIFT points, far too sparse to snap dense matches onto for PnP. - Per-image PnP only filters outliers: over near-planar terrain a single image's pose has a ~1.5 deg tilt/translation ambiguity. align_ir_to_eo solves one Sim3 (irworld <- eoworld) jointly over all images' inliers instead, so fused IR poses inherit the IR model's own multi-view relative geometry (fl09: sensor_from_rig spread 1.585 -> ~0.23 deg). - insert_ir_image attaches PnP-inlier keypoints that land within snap_px of a triangulated observation as observations of those EO 3D points (multimodal tracks), one per point per image. - refine_fused offers a fixed-EO bundle adjustment; off by default since per-image BA against the sparse cross-modal tracks re-adds the tilt wobble the joint alignment defeats. Tests run without torch (fake matcher over synthetic nadir geometry); matcher-contract tests skip unless vismatch is installed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-group calibration runs unchanged (it supplies the IR intrinsics and the warp initialization); with --fuse and both rgb+ir groups present, the IR images are then registered into the EO model (fusion.py), the boresight/sensor_from_rig/export re-run once on the fused reconstruction, and the same yamls/rig.json are written from it — IR extrinsics measured against the EO reference camera directly instead of relayed through the INS. IR cameras that fuse too few frames keep their two-boresight calibration. QC: each IR camera's mount delta vs the two-boresight solution is logged, a fusion_report.json (per-image match/inlier stats, model alignment, skip counts) lands next to the models, and the rig JSON gains a fused rgb+ir group and a fusion provenance block. The fused reconstruction is written to colmap_rig_fused/ for inspection. Without --fuse the outputs are byte-identical to before (verified on fl09 against the previous driver). Validated end-to-end on fl09: 633/835 IR images fused, alignment over 287,841 matches, fused IR mounts 0.19-0.22 deg from the two-boresight mounts — within the known INS-noise band, now measured from image evidence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ub.ensuredir for the fused-model directory, and functools.lru_cache in place of the hand-managed snap/image caches in fuse_ir_into_eo (the image cache's clear-at-16 was just a worse LRU). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four-angle cleanup pass (reuse/simplification/efficiency/altitude) over the fusion diff: - Generalize registration_homography's pixel homography with optional per-camera times and a ground plane (far-field remains the default), and delete fusion's warp_homography, which duplicated its grid/unproject/project/fit scaffold. - Drop dead bookkeeping: IrMatchResult.per_partner and the per-partner stats dict (written, never read), the unused num_snapped property, and EoPartner's image_id/name fields that shadowed its pycolmap Image. - Merge the lockstep matched/entries lists into one, and release each result's full depth-lifted point cloud once its inlier subset is copied into the alignment entry (it was held for all ~800 images). - Move the refine_ir_intrinsics-implies-run_ba rule into fuse_ir_into_eo so the knobs are self-consistent for every caller. - Extract _calibrate_group in the driver: the boresight -> extrinsics -> export -> records sequence was duplicated between the per-group loop and the fused path. - Trim the fusion module docstring to the two load-bearing rationales (depth lift, joint Sim3 vs per-image PnP). No behavior change: default-path outputs remain byte-identical on fl09 and the fused smoke run reproduces the pre-refactor results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Delete the half of colmap_processing that pycolmap replaced and that nothing outside the package imports: the hand-rolled COLMAP sqlite wrapper (database.py) and model reader (colmap_interface.py), the old calibration/SLAM drivers (calibration.py, dp.py, slam.py), the static ground-camera model (static_camera_model.py), the docker feature/matcher/mapper wrappers that rig.build_colmap_database replaced, and the one-off scripts built on all of the above. Every deleted script still used pre-namespace `colmap_processing.` imports, so none of it has been runnable since the kamera.* migration; the same was true of test/test_camera_models.py, which also depended on colmap_interface. What stays is the live sensor-model layer (camera_models.py's StandardCamera -- the exported product format -- with platform_pose, geo_conversions, rotations, image_renderer), whose future home is the geocam project, plus the VTK/geotiff rendering capability (vtk_util, world_models, geotiff, and the two geotiff scripts), kept for now with their import prefixes fixed to the kamera.* namespace so they import again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fold the prepare_flight step into the calibration driver: when a flight has no colmap_* workspace yet (or --raw-dir forces a restage), the raw imagery dir is auto-detected via flight_prep.find_raw_dir -- either the flight dir itself or its single subdirectory holding the view folders (both center_view/ and bare center/ layouts occur in the wild) -- and staged with the same frame-paring flags prepare_flight.py exposes (--spacing, --max-frames, --focal-px, --copy). Already-staged flights are untouched (verified byte-identical outputs on fl09). prepare_flight.py remains for staging-only runs, e.g. to inspect frame selection and overlap before committing to a calibration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Frame density and overlap are flight-planning concerns; the pipeline now stages every trigger that has a nav time and stops second-guessing the survey design. Removes select_by_spacing, the forward-overlap estimator, and the --spacing/--every/--max-frames/--focal-px/ --height-px flags from stage_flight, prepare_flight.py, and calibrate_rig.py (staging keeps just --raw-dir/--prefix/--copy); stage_flight no longer needs the nav provider at all. Also normalize utilities.stretch_constrast to a 1-99 percentile stretch (was 0.1-99.9) and fix its clamp typo (225 -> 255), matching deep_match.to_uint8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pycolmap's default num_threads=-1 spawns one extraction worker per core, each holding a full-resolution decode of the input image before the max_image_size downscale -- tens of GB of transient RAM on a many-core machine with 100 MP imagery (OOM-killed in the field), and worse if pycolmap silently fell back to its CPU build, where each worker also carries the 2x-upsampled 11-octave SIFT pyramid. A few feeder threads already saturate the GPU; cap at 8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.