feat: InstantMesh reconstruction, browser physics gallery, and AI object identification - #31
Merged
Merged
Conversation
… gallery, AI identification
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reconstruction moves to InstantMesh, the viewer runs real physics on the mesh it
just built, and the agent names the object and picks the test worth watching.
Measured on the same car photograph, before and after:
InstantMesh as a third pipeline
PIPELINE_NAME=stub | triposr | instantmesh. TripoSR is untouched and still in theimage, so a bad result is one env var away from rollback with no rebuild. Zero123++
generates six view-consistent images, an LRM reconstructs from all six — where
TripoSR predicted geometry from one photograph and invented the other five sides.
nvdiffrast is stubbed, not built.
src/models/lrm.pyimports it at module scopebut
extract_mesh(use_texture_map=False)returns early and never reaches it. Thatremoves the CUDA extension build entirely — the one that failed twice on TRELLIS.
The stub raises on attribute access rather than returning a mock, so if that path is
ever taken it fails loudly instead of shipping a wrong mesh.
Everything pinned: InstantMesh
08822c52, weightsb785b4ec, Zero123++2da07e89,its custom pipeline
983e66d2. The pipeline file lives in a different HF repo fromits weights; both are baked in and loaded
local_files_only, so nothing is fetched atstartup.
Five defects found only by running it
from_pretrainedpipeline.pyneedstrust_remote_code=True. Not a trust decision here: pinned at build time,local_files_only, unchangeable in the containerModuleNotFoundError: xatlasmesh_util.pyimportsxatlasandcv2at module scope for the UV path. Installed rather than stubbed — a real module cannot be subtly wrong the way a stub of someone else's API caninstant-nerf-large.yamlsetsencoder_model_name: facebook/dino-vitb16and callsfrom_pretrained. TripoSR only ever needed the config, so only the config was bakedpipelineInfo.nameenum gainedinstantmeshbut only reconstruction was redeployed. A contract change means redeploying every service that validates itcontract-mismatchwith the offending fieldMesh quality
formula reads "not measured" as "worst possible" — at 45% weight. Now sampled on a
48³ grid in the triplane's own space.
generated views and reconstructed as geometry: walls either side of the subject.
volume becomes mass becomes force.
simplify_quadric_decimationdiscards colour and returns a uniform white mesh.A CPU texture bake (xatlas + numpy rasterisation, colour sampled at texel resolution)
is included but disabled at
TEXTURE_RESOLUTION=0. It works — 53× the colourdetail — but UV unwrapping splits vertices at chart seams, which makes the surface
topologically open and drops watertightness to zero. For a physics system that is the
wrong trade: mass comes from volume, and volume needs a closed surface.
Browser physics gallery
Rapier runs in the viewer, on the mesh just reconstructed, with the engine and seed
the physics service uses. Four tests — lateral push, drop, impact, load — each with
its own verdict, tilt, drift and step count, and the object moves in the viewport.
packages/scenegains an optionalonPoseobserver called after every step. Theserver never passes one, so the result and its determinism digest cannot depend on
whether anyone watched. All 28 parity tests pass unchanged.
Impact needed no contract change: a crash is a drop with enough height to matter.
AI identification
POST /v1/identifyon the agent, proxied by the web. Gemini Flash returns the object,its size in metres, its material, and which test is worth watching first with the
geometry it reasoned from. The gallery opens on that test.
Read-only and unpersisted — the scan page has a mesh and no job behind it, so this
answers the viewer rather than pretending a decision was made. It runs after the mesh
is on screen and every failure returns null: a slow model must not cost a scan.
The agent also now states the object's scale to the reconstruction service.
assumedLongestDimensionMeterswas already in the request contract and was neversent, so every object fell back to 0.30 m. Same bottle, same mesh, same seed:
0.30 m → 0.23 m moved tilt 94.33° → 105.80°.
Viewer
ACES tone mapping, image-based lighting, three-point rig with soft shadows, PBR.
glTF declares
COLOR_0linear while a reconstruction writes sRGB bytes — read aslinear they came out dark, which is now corrected once and guarded.
§11's confidence-as-motion is cut. It was built, and it read as a broken viewer
rather than as a signal. The number and its four components already state the
uncertainty in words.
Gates
123 reconstruction tests · 168 agent · 68 contract · 28 scene · 9 web ·
ruff·mypy·pip-audit --strict· no contract drift