HaploSR-MV — posed multi-view 3D reconstruction
HaploSR-MV turns several posed photos of one object into a single 3D mesh
that is measurably closer to ground truth than a single-image reconstruction.
It wraps a frozen TripoSR (MIT) with a differentiable multi-view refinement
stage: the triplane scene code is optimized so its rendered silhouettes match
every view, while an anchor regularizer preserves the sharp detail of the seed
view. On device the camera poses come for free from ARKit, which removes the
pose-free ambiguity that makes multi-view reconstruction hard.
Base model TripoSR is MIT-licensed, so HaploSR-MV has no commercial
restrictions — no attribution requirements, no revenue caps. It's built
entirely on permissively-licensed components.
Single photo in, 3D mesh out
HaploSR-MV seeds from a single-image TripoSR reconstruction, then refines it with
additional posed views. Here's what the single-photo seed already produces on
device (multi-view refinement then sharpens it — see the +33% gate below):
Validated result (gate)
Measured on a 4-object subset with elevation-diverse views, ICP-aligned
scale-free metrics (Chamfer ↓ better, F-score ↑ better):
| object | Chamfer base→refined | F-score base→refined |
|---|
| Duck | 0.266 → 0.168 (+36.8%) | 0.155 → 0.203 (+30.4%) |
| Avocado | 0.033 → 0.027 (+19.0%) | 0.723 → 0.816 (+12.8%) |
| BoomBox | 0.076 → 0.064 (+15.6%) | 0.330 → 0.418 (+26.7%) |
| WaterBottle | 0.039 → 0.019 (+52.3%) | 0.603 → 0.914 (+51.6%) |
| mean | +32.9% | +29.8% |
4/4 objects improved on both metrics; no regressions.
The one capture rule that matters
Views must be elevation-diverse, not just an azimuthal orbit. Azimuth-only
silhouettes carve a generalized cylinder and cannot constrain top/bottom
geometry — in early testing that regressed F-score. Guide the capture UX to vary
tilt as well as angle (a natural handheld arc around the object does this).
Usage
1from haplosr_mv.pipeline import HaploSRMV, RefineConfig, View
2from PIL import Image
3
4pipe = HaploSRMV(RefineConfig(steps=150, mc_resolution=256))
5views = [
6 View(image=Image.open("v0.png"), azimuth_deg=0, elevation_deg=0), # seed
7 View(image=Image.open("v1.png"), azimuth_deg=55, elevation_deg=28),
8 View(image=Image.open("v2.png"), azimuth_deg=125, elevation_deg=-22),
9 # ... ARKit path: pass c2w=<4x4 numpy> instead of azimuth/elevation
10]
11mesh, _ = pipe.run(views)
12mesh.export("out.glb")
Or the CLI: python run.py --demo (A/B on a bundled object), or
python run.py --views-dir <dir> with a poses.json.
How it works
- Seed — TripoSR encodes the canonical view (az=0, el=0) into a triplane.
- Refine — with the network frozen, Adam optimizes the triplane against a
stochastic silhouette loss over all posed views (+ optional photometric),
plus an anchor term
‖code − seed‖² (weight 2e-3) that keeps seed detail.
150 steps, lr 8.0 (the triplane std is ~377, so it needs a large lr),
1500 rays/view at 256².
- Extract — marching cubes on the refined triplane.
Camera convention exactly matches TripoSR's get_spherical_cameras
(x-back, y-right, z-up; seed at az=0 el=0), so ARKit c2w poses drop in directly.
Limitations
- The refiner is an optimization loop (~seconds–minutes on Apple GPU/MPS),
best run server-side or as a "premium quality" path — not real-time. For
real-time on-device fusion use the feed-forward
sf3d_mv_encoder.onnx.
- Silhouette-dominant loss: needs reasonable foreground masks per view (alpha,
or a segmenter such as BiRefNet/rembg).
- Inherits TripoSR's single-view seed quality as a starting point.
License
MIT (this refinement method + code). Base weights: TripoSR, MIT
(stabilityai/TripoSR). No attribution or revenue restrictions.
