Add a foundation-stereo option for warping detections between stereo cameras - #1875
Add a foundation-stereo option for warping detections between stereo cameras#1875mattdawkins wants to merge 1 commit into
Conversation
…cameras The existing client-side transfer matches each point by NCC template matching along its epipolar curve, which needs the source patch to be photometrically matchable in the other view. Where that fails -- obstructed viewpoints, repetitive substrate, low contrast -- it mismatches or declines. Adds a second correspondence method that runs a Fast-FoundationStereo ONNX export instead: rectify the pair once per frame, run a dense disparity network, and read each point's shift out of the disparity map. Both methods satisfy a shared StereoMatcher interface, so box/line/polygon warping, measurement and bulk transfer are unchanged either way, and a Correspondence method dropdown under Track Settings -> Stereo Settings picks between them. The foundation model is not bundled -- the exports run ~100 MB -- so it is supplied by URL and the method no-ops with a warning when none is served. NCC remains the default.
|
A 100MB stereo file shouldn't be bundled in the app or in the repo. I think for the Web version it should be in a viame.kitware.com location and there should be a configuration option for the server that points at either the direct URL or girder Item. Maybe it is a place where we can configure multiple models in the future if we start using moe onnx models. For the desktop version I would probably have that onnx model embedded inside of the VIAME install and the express server in electron would serve that to the client when requested. It could fall back to using the web version if it has an internet connection Then the client when requested the first time would download and cache it for use. I think this makes it a bit easier in the future for deployment of other models as well if we are going to rely on onnx web runtimes for other functionality. |
What
Adds a second stereo correspondence method — a Fast-FoundationStereo ONNX
export — alongside the existing NCC template matcher, with a dropdown under
Track Settings → Stereo Settings → Correspondence method to pick between
them. NCC remains the default.
Why
The existing matcher correlates the source patch along its epipolar curve, so it
needs that patch to be photometrically matchable in the other view. Where that
breaks down — obstructed viewpoints, repetitive substrate, low contrast — it
either mismatches or declines to answer.
A dense disparity network doesn't depend on patch correlation, and it costs one
pass per frame regardless of how many points are warped, so bulk-warping a whole
camera amortises well.
How
Both methods satisfy a shared
StereoMatcherinterface, so box / line / polygonwarping, measurement and bulk transfer are untouched and nothing downstream
branches on the choice.
The foundation path:
rectify.ts, ported fromOpenCV
cvStereoRectify), sized to the network's input resolution.pixel and bilinear-sampling — rectify and resize fused, so cost scales with
the network's resolution, not the frame's.
x,unrectify into the target image.
Step 4 pools rather than reading one pixel deliberately: a head or tail tip is a
couple of pixels wide at the network's working resolution, so the disparity
exactly at the tip is frequently the background's.
The network has no confidence channel, so
scoreis the fraction of the pooledwindow carrying a finite positive disparity; a match is accepted when that
clears a threshold and the implied disparity is inside the configured search
range — the same range that already bounds the NCC search.
The model is not bundled
Fast-FoundationStereo exports run ~100 MB, well past what belongs in the repo.
It's supplied by URL:
foundationModelSpecmust match the export — the graph fixes its inputresolution, and the sidecar
.yamlgives it asimage_size: [H, W]. With nomodel served, selecting the method logs a warning and the warp no-ops, the same
way a missing calibration already does.
Testing
tests/rectify.spec.ts(new, 6 cases): Rodrigues round-trip, orthonormality ofthe rectifying rotations, the defining rectification property (a 3D point lands
on the same row in both rectified views), disparity positive and decreasing
with range, and pixel round-trip through rectify/unrectify with and without
distortion.
Not tested:
StereoFoundationMatcherend-to-end, which needs the ~100 MBmodel the repo doesn't carry. The geometry it rests on is covered above, but the
network call, the disparity pooling and the dropdown have not been exercised
against a real export in a running viewer. Worth a look from someone who can
point it at a model.