Skip to content

feat: InstantMesh reconstruction, browser physics gallery, and AI object identification - #31

Merged
Mizunandayo merged 1 commit into
mainfrom
feat/instantmesh-and-browser-physics
Aug 31, 2026
Merged

Mizunandayo merged 1 commit into
mainfrom
feat/instantmesh-and-browser-physics

Conversation

@Mizunandayo

Copy link
Copy Markdown
Owner

Reconstruction moves to InstantMesh, the viewer runs real physics on the mesh it
just built, and the agent names the object and picks the test worth watching.

Measured on the same car photograph, before and after:

Before After
Reconstruction confidence 39.6% 94.4%
Field decisiveness 0.0% 99.4%
Foreground quality 48.0% 87.6%
Watertight no yes

InstantMesh as a third pipeline

PIPELINE_NAME=stub | triposr | instantmesh. TripoSR is untouched and still in the
image, so a bad result is one env var away from rollback with no rebuild. Zero123++
generates six view-consistent images, an LRM reconstructs from all six — where
TripoSR predicted geometry from one photograph and invented the other five sides.

nvdiffrast is stubbed, not built. src/models/lrm.py imports it at module scope
but extract_mesh(use_texture_map=False) returns early and never reaches it. That
removes the CUDA extension build entirely — the one that failed twice on TRELLIS.
The stub raises on attribute access rather than returning a mock, so if that path is
ever taken it fails loudly instead of shipping a wrong mesh.

Everything pinned: InstantMesh 08822c52, weights b785b4ec, Zero123++ 2da07e89,
its custom pipeline 983e66d2. The pipeline file lives in a different HF repo from
its weights; both are baked in and loaded local_files_only, so nothing is fetched at
startup.

Five defects found only by running it

Defect Cause
Startup probe failed at from_pretrained Vendored pipeline.py needs trust_remote_code=True. Not a trust decision here: pinned at build time, local_files_only, unchangeable in the container
ModuleNotFoundError: xatlas mesh_util.py imports xatlas and cv2 at module scope for the UV path. Installed rather than stubbed — a real module cannot be subtly wrong the way a stub of someone else's API can
DINO weights missing instant-nerf-large.yaml sets encoder_model_name: facebook/dino-vitb16 and calls from_pretrained. TripoSR only ever needed the config, so only the config was baked
Web rejected a valid response The pipelineInfo.name enum gained instantmesh but only reconstruction was redeployed. A contract change means redeploying every service that validates it
"unreachable" hid a contract mismatch for three rounds The catch block treated any non-timeout throw as a network failure. Contract violations now report contract-mismatch with the offending field

Mesh quality

  • Field decisiveness was 0.0% because the adapter passed an empty array, and the
    formula reads "not measured" as "worst possible" — at 45% weight. Now sampled on a
    48³ grid in the triplane's own space.
  • Zero123++ needs a white background. TripoSR's mid grey was carried into all six
    generated views and reconstructed as geometry: walls either side of the subject.
  • Taubin smoothing, not Laplacian — the alternating pass preserves volume, and
    volume becomes mass becomes force.
  • Decimation to 80k faces, with vertex colour resampled from the original surface:
    simplify_quadric_decimation discards colour and returns a uniform white mesh.

A CPU texture bake (xatlas + numpy rasterisation, colour sampled at texel resolution)
is included but disabled at TEXTURE_RESOLUTION=0. It works — 53× the colour
detail — but UV unwrapping splits vertices at chart seams, which makes the surface
topologically open and drops watertightness to zero. For a physics system that is the
wrong trade: mass comes from volume, and volume needs a closed surface.

Browser physics gallery

Rapier runs in the viewer, on the mesh just reconstructed, with the engine and seed
the physics service uses. Four tests — lateral push, drop, impact, load — each with
its own verdict, tilt, drift and step count, and the object moves in the viewport.

packages/scene gains an optional onPose observer called after every step. The
server never passes one, so the result and its determinism digest cannot depend on
whether anyone watched. All 28 parity tests pass unchanged.

Impact needed no contract change: a crash is a drop with enough height to matter.

AI identification

POST /v1/identify on the agent, proxied by the web. Gemini Flash returns the object,
its size in metres, its material, and which test is worth watching first with the
geometry it reasoned from. The gallery opens on that test.

Read-only and unpersisted — the scan page has a mesh and no job behind it, so this
answers the viewer rather than pretending a decision was made. It runs after the mesh
is on screen and every failure returns null: a slow model must not cost a scan.

The agent also now states the object's scale to the reconstruction service.
assumedLongestDimensionMeters was already in the request contract and was never
sent, so every object fell back to 0.30 m. Same bottle, same mesh, same seed:
0.30 m → 0.23 m moved tilt 94.33° → 105.80°.

Viewer

ACES tone mapping, image-based lighting, three-point rig with soft shadows, PBR.
glTF declares COLOR_0 linear while a reconstruction writes sRGB bytes — read as
linear they came out dark, which is now corrected once and guarded.

§11's confidence-as-motion is cut. It was built, and it read as a broken viewer
rather than as a signal. The number and its four components already state the
uncertainty in words.

Gates

123 reconstruction tests · 168 agent · 68 contract · 28 scene · 9 web ·
ruff · mypy · pip-audit --strict · no contract drift

@Mizunandayo
Mizunandayo merged commit 2de6a73 into main Aug 31, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant