fix(rerun): stabilize Go2 3D detection previews - #3571
Conversation
github-actions
Bot
added
the
first-time-contributor
Greptile SummaryThis change adds fixed-size letterboxed previews for successful 3D detections and routes detection images and point clouds through shared Go2 visualization slots. The viewer-disabled configuration is not preserved: the detection blueprint directly creates a Rerun bridge even though the The reported shared-slot mapping issue was dropped after an executed check exercised Confidence Score: 4/5Not safe to merge as-is because a deployment configured with no viewer can still start the Rerun bridge. A focused executable source probe confirmed the viewer-none baseline, the unconditional bridge declaration, and the coordinator startup path. The shared-slot topic behavior was also directly exercised and behaved correctly. Files Needing Attention: dimos/robot/unitree/go2/blueprints/smart/unitree_go2_detection.py needs its Rerun bridge construction to respect the selected viewer backend.
 T-Rex LogsWhat T-Rex did
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| unitree_go2, | ||
| # Replaces the RerunBridgeModule already present in unitree_go2 while | ||
| # leaving its single vis_module bundle and websocket modules intact. | ||
| RerunBridgeModule.blueprint(**detection_rerun_config), |
There was a problem hiding this comment.
The base Go2 visualization configuration omits RerunBridgeModule when the viewer backend is none, but this blueprint unconditionally adds a replacement bridge here. Since active blueprint modules are started by the coordinator, a deployment that explicitly disables the viewer still starts Rerun publishing. Build the detection-specific bridge through the viewer-aware visualization configuration, or add it only when the resolved backend is Rerun.
Artifacts
Authored viewer-none blueprint introspection script
- Dependency-free AST probe used to inspect the exact viewer-none branch, detection bridge declaration, and coordinator startup path; it provides the executable comparison method.
Inherited Go2 blueprint behavior with viewer none
- Executed inherited-case capture shows the viewer-none branch includes WebsocketVisModule and zero inherited Rerun bridges, establishing the baseline.
Detection Go2 blueprint behavior with viewer none
- Executed detection-case capture shows the direct bridge at line 59 and the coordinator startup call, confirming viewer-none does not suppress this bridge.
There was a problem hiding this comment.
this seems legit and it bugs me a bit, our impl of how we add viewers is a bit messy but can you sort something out? other then that tested on my end ready to merge
There was a problem hiding this comment.
The core issue: vis_module() in unitree_go2_basic.py is viewer-aware (skips RerunBridgeModule for "none"), but the detection blueprint calls RerunBridgeModule.blueprint() unconditionally, bypassing that guard.
The clean fix is to mirror the same guard from vis_module():
| RerunBridgeModule.blueprint(**detection_rerun_config), | |
| detection_rerun_config = { | |
| **rerun_config, | |
| "topic_to_entity": _detection_topic_to_entity, | |
| } | |
| from dimos.core.global_config import global_config | |
| def _detection_vis_modules() -> list: | |
| """Return bridge module only when the configured viewer is Rerun.""" | |
| if global_config.viewer == "rerun": | |
| return [RerunBridgeModule.blueprint(**detection_rerun_config)] | |
| return [] | |
| unitree_go2_detection = ( | |
| autoconnect( | |
| unitree_go2, | |
| # Replaces the RerunBridgeModule already present in unitree_go2 while | |
| # leaving its single vis_module bundle and websocket modules intact. | |
| # Guarded so deployments with viewer=none skip this entirely. | |
| *_detection_vis_modules(), |
That said, the messiness you're pointing at is real: vis_module() encapsulates the backend switch cleanly, but detection_rerun_config needs topic_to_entity which vis_module() doesn't expose as a first-class parameter. The better longer-term fix is to thread topic_to_entity (and other bridge-specific overrides) through vis_module() so that all blueprints go through the same guarded factory rather than calling RerunBridgeModule.blueprint() directly. That would be a small addition to vis_module's rerun_config dict — RerunBridgeModule already accepts it — but it's a broader change across all robot blueprints.
For now the guard above unblocks the merge. Happy to draft the vis_module() refactor as a follow-up if useful.
There was a problem hiding this comment.
The fix is simple — swap the direct RerunBridgeModule.blueprint() call for vis_module(), matching the pattern in unitree_go2_basic.py:
Import changes:
| RerunBridgeModule.blueprint(**detection_rerun_config), | |
| +from dimos.core.global_config import global_config | |
| from dimos.robot.unitree.go2.blueprints.basic.unitree_go2_basic import rerun_config | |
| from dimos.robot.unitree.go2.blueprints.smart.unitree_go2 import unitree_go2 | |
| from dimos.robot.unitree.go2.connection import GO2Connection | |
| -from dimos.visualization.rerun.bridge import RerunBridgeModule | |
| +from dimos.visualization.vis_module import vis_module |
Call site (replacing the RerunBridgeModule.blueprint(...) line):
| RerunBridgeModule.blueprint(**detection_rerun_config), | |
| + # Replaces the vis_module bundle already present in unitree_go2 while | |
| + # leaving its websocket module intact. | |
| + vis_module( | |
| + viewer_backend=global_config.viewer, | |
| + rerun_config=detection_rerun_config, | |
| + ), |
vis_module() already has the match viewer_backend dispatch — "rerun" adds the bridge, "none" skips it entirely. No other changes needed.
2 participants
Contribution path
Problem
unitree-go2-detectionpublishes cropped detection images to Rerun. Because crop dimensions vary between frames, sending them as JPEGrr.EncodedImagevalues under the same entity path makes Rerun treat them as a continuous video stream whose encoding properties change over time.In our reproduction, the replay can continue running if these dynamic crop entities are not inspected in the Rerun Viewer. However, selecting one of the affected image entities and displaying it in the Viewer panel triggers the rendering path and can cause the Rerun Viewer process to panic.
The issue can be reproduced by running:
And click following area,
Rerun Viewer screenshot:
Key Rerun log:
There is also a separate slot-alignment problem in the existing
Detection3DModulevisualization:detected_image_Nanddetected_pointcloud_Ncan refer to different detections.This is a slot-compaction mismatch rather than a temporal synchronization problem.
A runnable reproduction is available on the following branch:
https://github.com/HaroldGin931/dimos/tree/chore/repro-go2-2d-3d-crop-mismatch
Run it with:
The reproduction compares hashes from the original 2D detection crops with the crops associated with the compacted 3D results. It pauses Rerun on the first mismatch and prints output similar to:
In this example, 2D display slot 0 and 3D display slot 0 have different image hashes. The 3D result shown in slot 0 originated from 2D detection slot 1 because an earlier detection was discarded during 3D projection.
After Rerun pauses, the corresponding 2D and 3D slot entries can be inspected in the Viewer. Their displayed crops are visibly different, demonstrating that the original visualization incorrectly treated the two independently ranked slots as a matching image and point-cloud pair.
This PR only fixes the Go2 visualization path and does not change the underlying 2D-to-3D filtering behavior. We do not currently have access to a G1 or other affected robots for validation, so this PR makes the smallest change that can be tested end-to-end on Go2. Other Blueprints publishing variable-sized
detected_image_*streams may require a separately tested follow-up.Solution
Publish a crop from each successful 3D detection alongside its corresponding point cloud.
Add
detected_3d_image_0/1/2outputs so the image and point cloud originate from the sameDetection3DPC.Letterbox each 3D crop to a fixed 320×320 preview while preserving its aspect ratio.
Map each crop and point-cloud pair to the same Rerun entity, for example:
This allows selecting one slot in Rerun and inspecting both its image and point cloud.
Disable the inherited raw 2D crop previews in the Go2 detection Blueprint because those slots are ranked before unsuccessful 3D projections are removed.
Reuse the existing Go2 visualization bundle and Rerun layout rather than adding a second
vis_module().Keep the global
Image.to_rerun()behavior unchanged so fixed-resolution camera streams continue usingrr.EncodedImage.How to Test
Verify that:
AI assistance
OpenAI Codex with GPT-5 assisted with codebase analysis, debugging, implementation, validation, and drafting this PR description. The changes were manually reviewed and the replay behavior was manually verified.
Checklist