Conversation
`--episode-index` is now optional. Omitting it loads the whole dataset and gives each episode its own tab, rooted at an `episode_N/` entity subtree, so episodes no longer overwrite one another on shared entity paths. Two timelines make the multi-episode view usable: `episode` selects an episode, and `episode_frame` scrubs every episode in parallel at its own frame N, while `frame_index` still walks the dataset end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a headless MuJoCo sim of an SO-101 arm picking up cubes and stacking them, plus a scripted expert that solves it, a Gymnasium env that follows LeRobot's `pixels`/`agent_pos` convention, and a recorder that writes the expert's demonstrations as a LeRobotDataset. The SO-101 description is fetched from TheRobotStudio/SO-ARM100 (pinned to a commit) and cached, rather than vendored -- its meshes are ~16 MB. Two patches are applied to it at load time, both necessary: - Both jaws collide as convex meshes, and their hulls overlap by 13.6 mm at every gripper angle, so the space between the fingers is solid to the collision engine and nothing can be grasped. The jaw meshes are demoted to visual-only and replaced with box pads on the real finger faces. - The upstream wrist-camera variant models the mount but declares no `<camera>`. One is added, offset past the module's front face. Measured expert success (random layouts): 100% for 2/3/4 blocks at 20 mm (20/60/30 episodes), 100% at 25 mm, 95% at 30 mm. ~0.2 s wall per episode. Two properties of this arm drove most of the design and are documented in the README, since both fail silently rather than loudly: - Only one jaw moves, so the centre of the jaw opening shifts with the gap, and opening the jaws frees a held block on one side only. - Holding the gripper vertical costs two of five joints, leaving a workspace that tops out near z = 0.09 m -- far below the arm's ~0.35 m reach. Out-of- envelope waypoints don't raise; the IK returns a non-converged guess and the arm quietly goes elsewhere. `calibrate_gripper.py` re-derives the gripper constants from the model and `workspace.py` prints the reachable envelope, so neither has to be guessed. Tests pin the constants against the live model and assert the fingertips never touch the table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-ran the validation matrix after cleanup: 100/100 on the default 3-block 20 mm config, and larger samples for the other variants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ng docs `policy.push_to_hub` defaults to true, so the copy-pasteable training command would have attempted an upload at the end of a run. Pass `--policy.push_to_hub=false` and say how to opt back in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two SceneConfigs that differ wrote to the same path, so an env created with a different block size would overwrite the file another env had just loaded from and silently pick up the wrong geometry. Name the file after a hash of the generated MJCF; identical configs still share one file so vectorised envs reuse it. Also drops unused config fields (`SceneConfig.seed`, the `extras` placeholders) and the redundant `stack_order()` accessor, which duplicated `info["block_order"]`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Cartesian interpolator took its starting position from the live site pose but hardcoded the starting yaw to zero, so the first command of an episode asked for a wrist rotation the arm was not at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…king expert A 300-episode sweep turned up one failure (seed 220): a clamped block crept ~8 mm in the jaws during the long lateral carry, so the place was aimed with a stale carry offset and landed 9.7 mm off centre -- just past the stability tolerance, and it toppled. The offset was only measured once, right after the jaws closed. Measure it again at the end of the transit move so slip during the carry is corrected. 300/300 after the fix, and 2-block, 4-block and 25 mm variants all stay at 100% over 40-60 episodes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-101 Adds a second stackable piece alongside the cube: the Duplo 2x2 brick at library.ldraw.org/parts/6083 (part 3437), and a `duplo_yellow_on_blue` task that puts a yellow brick on a blue one. - ldraw.py resolves the official part file and its subparts/primitives into a triangle mesh (BFC-aware, so the hollow underside is not inside out). It is used for rendering only. - pieces.py builds the collision shape from primitives: top plate, four walls, eight locating ribs and four studs, with the cavity open underneath. A mesh would collide as its convex hull and fill the cavity. Dimensions come straight from the LDraw geometry (32 x 32 x 19.2 mm, studs 9.6 mm dia). A dropped brick seats over the studs (28.78 mm vs 28.80 expected, not the 33.2 mm of one perched on top) and the ribs hold it within 0.4 mm. - The expert turns the carried brick to match the one below and tracks the carried pose in the grasp-site frame so the turn moves it predictably. - Success for interlocking pieces checks yaw and a 1.5 mm z tolerance, well under the 4.4 mm stud height, so a perched brick is not scored as seated. The grasp needed a real fix to the gripper model. The jaw gap was taken as the closest distance between the pads, but the moving jaw is a hinged finger: past a few degrees its pad tilts, leaning in at the top and splaying out at the tip, so the closest approach is up near the hinge while an object on the table is pinched much lower. For a 32 mm brick the jaws at brick height were ~9 mm wider than commanded and closed on air (0/12). `jaw_opening(angle, depth)` now computes the gap at a given depth from the pad boxes, and the expert sets every gap at the block's top edge. The old table overstated the angle needed by about 2x (hold: 18.6 deg commanded, 9.1 deg correct). Duplo: 12/12 seated (0.2-0.5 mm, 0.0 deg). Cubes unchanged: 40/40. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eds) A 100-seed sweep of the Duplo task passed 93. The seven failures had four unrelated causes; all fixed, 100/100 afterwards, cubes unchanged (60/60, 4-block 20/20). - Success fired on a single frame mid-release. A block the jaws let go of is briefly free of both pads while it drops the last millimetre, then settles back against the fixed jaw, which does not open. Ending the episode there froze the arm mid-release and the final score counted the brick as held. Success must now hold for 10 consecutive control steps. - The start pose hung the open moving jaw 28 mm above the table and 29 cm out, over the spawn area, and knocked a brick that spawned beneath it. The start pose is now solved by IK: fingers down, 9 cm up, every joint >= 0.11 rad inside its limits. - The pick was planned from the block pose at the start of the sequence. It is now re-read from directly above the block before descending, so any nudge on the way is absorbed instead of a pad landing on the studs. - Of the four equivalent place yaws the expert took the one needing the least rotation, which on the far side of the table was 8 deg past the wrist_roll limit. The IK clamped silently and the brick went down 18 deg off. Candidate yaws are now checked for reachability first. Also: the agent_pos observation bounds came from the actuator ctrlrange, which the MJCF rounds inside the joint range; a joint at its (soft) limit read a few microradians outside the space. Bounds now use the joint range plus a margin. The demo lets the expert finish and folds the arm away before judging and filming the result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…task - tests/test_so101_duplo.py (16 tests): the rendered mesh matches LDraw 3437 and winds outward; an aligned brick seats at exactly one body height and is held by the studs against a sideways shove; a brick rotated 45 deg perches on the studs and is *not* scored as stacked; success must persist; the jaw gap grows with depth once the moving pad tilts, and the gap->angle inverse round-trips; the expert seats the yellow brick fully on seeds including 43 and 96 (the two that failed before the last fixes), and with the solid collision model. - README: the Duplo task, how the brick is modelled (render mesh from LDraw, collision from primitives, seats but does not clutch), the height-dependent jaw gap, and the updated limitations -- it previously said interlocking bricks were not modelled. - record_dataset.py: --task, and each frame now stores the task's own instruction instead of a hardcoded cube sentence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…for it Held-out cube seeds turned up two failures (1035, 1192 of 1000-1199). In both, the IK -- asked for position *and* jaw yaw, and seeded from the previous solution -- ended up pinned against the wrist_roll limit because the requested turn went the wrong way round. It then either stalled 80 mm short of the block while a fresh solution sat on the other wrist branch, or reported the waypoint reached with the jaws ~85 deg off and finished the turn during the descent, sweeping the cube aside. With the fingers pointing down the grasp site sits on the wrist-roll axis, which is also the approach axis: turning wrist_roll moves neither (0.00 um, ~1e-16 over 200 poses) and changes the jaw yaw exactly 1:1. So the IK now solves only position + fingers-down, and wrist_roll is set directly, interpolated in joint space to the roll that gives the requested yaw. Of a square piece's four equivalent yaws, pick and place use the one inside the roll range with the least wrist travel. Position tracking can no longer be blocked by the wrist. Results: cubes 1000-1199 200/200 (was 198), 2000-2199 200/200, 0-59 60/60, 4-block 40/40; Duplo 0-399 400/400. Also makes every generated file (patched arm MJCF, scene, LDraw mesh and downloads) an atomic write-if-changed with per-process temp names: several env processes build and load the same files at once, and a plain overwrite could hand one of them a half-written model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eeds Asserts that wrist_roll moves neither the grasp site nor the approach axis and turns the jaw yaw 1:1 (what the expert now relies on), and runs the expert on the two held-out seeds that used to fail. README success table now reports held-out seed ranges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds camera/calibration error and gear backlash (realism.py, --realism realistic), a snap-fit Duplo brick that needs a push to seat (--collision snap), and a set-down / relative-align / spiral-press placement strategy for it. Realistic error, seeds 0-49: exact placement 18/50, plain press 22/50, spiral press 32/50; perfect information 30/30. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The snap fit now follows how a Duplo brick actually holds: the centre tube under each brick (LDraw stud8) meets the four hollow studs 1.6 mm above the bottom edge, so a brick set down sits ~2 mm up, loose, until pushed; the grip that the flexing studs give is applied as dry friction (~3.5 N in, ~3.6 N out). Ribs touch the studs as in LDraw 3437, stud rims are only slightly rounded, and brick-on-brick friction is ABS-on-ABS (0.2). The expert places a brick as a person does: set it on, feel for it through the servo loads (Present_Load mapped through the Jacobian), slide it over if it jammed off-centre, push to ~8 N, look, push again if it is not home, open the jaws before leaving. Presses at 45 deg so a fingertip cannot wedge between studs; an empty pick is retried. Also: reset builds a fresh MjData and recomputes the fit, so an episode no longer depends on the one before it. Realistic error: 149/150 (the old place-and-release: 0/50). Perfect information: 30/30 (place-and-release: 0/30). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brick, table and fingertip contacts are now stiff (an 8 N press sinks one brick into another by ~0.03 mm, was up to 1.25 mm into the table), and the LDraw mesh is written with crease-angle normals so flat faces no longer shade as if bent. The clutch stays the only compliance. Stiffer contacts exposed three expert problems, fixed: ease off a push slowly (released at once, contact recovery popped the brick off); feel for a pick collision through the servo loads, only near the block and after a fast pre-descent, and retry wider instead of pushing until timeout; reset the kinematics scratch pose so episodes are bit-for-bit independent. Realistic 148/150, perfect information 30/30; studded 400/400, cubes 460/460 and 40/40 (episodes ~1 s longer from the slow final descent). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / Motivation
A headless MuJoCo simulation of the SO-101 arm picking up and stacking pieces: plain cubes, and a real Duplo 2x2 brick (LDraw 3437) that has to go down over the studs of the one below. It comes with a scripted expert that solves both tasks, a Gymnasium env that follows LeRobot's
pixels/agent_posconvention (so it plugs intolerobot-train/lerobot-evalvia EnvHub with no adapter), and a recorder that writes the expert's demonstrations as a LeRobotDataset whose feature names and action order match a realso101_followerrecording.The point is to be able to generate demonstrations and train an imitation policy for the SO-101 without the hardware, and to prototype on the same task (Duplo yellow-on-blue) that we collect on the real arm.
Related issues
What changed
All of the simulation lives in
examples/so101_block_stacking/; the only library change is the dataset viewer.scene.py,env.py,assets.py): SO-101 scene withfront/side/top/wristcameras, the wrist one at the real mount. The arm description is fetched fromTheRobotStudio/SO-ARM100at a pinned commit and cached, not vendored (~16 MB of meshes). Two patches are applied at load time, both required: the jaw meshes' convex hulls overlap by 13.6 mm at every angle (so nothing could be grasped) and are replaced with box pads on the real finger faces; and the upstream wrist-camera variant declares no<camera>, so one is added.ldraw.py,pieces.py): the render mesh is resolved from the official LDraw part (BFC-aware winding); collision is built from primitives (top plate, walls, ribs, studs) because MuJoCo would collide a mesh as its convex hull and fill the cavity.--collision studded(default) seats and locks,solidis a plain box for comparison, andsnapmodels the centre-tube fit so a brick has to be pushed in.expert.py,kinematics.py): IK solves position + fingers-down only, and wrist_roll sets the jaw yaw directly (it moves neither the grasp site nor the approach axis, so this is exact). It re-measures the carried piece after transit and re-reads the pick pose from directly above the piece. For the snap-fit brick it sets the brick down, reads contact through the servo loads, re-centres if jammed, and presses to ~8 N.jaw_opening(angle, depth)computes it from the pad geometry; the old closest-point table overstated the angle needed by ~2x, and closed on air for a 32 mm brick.realism.py,--realism realistic): camera/calibration error and gear backlash.record_dataset.py(--task, keeps only successes by default),demo.py(video/viewer),calibrate_gripper.pyandworkspace.py(re-derive the gripper constants and print the reachable envelope instead of guessing them).lerobot-dataset-viz:--episode-indexis now optional. Omitting it loads every episode, one tab each under anepisode_N/entity subtree, with anepisode_frametimeline to scrub all episodes in parallel. Backward compatible: passing--episode-indexbehaves as before.No breaking changes.
Measured expert success, random layouts, no cherry-picking (seeds 0-99 were used during development, the rest are held out):
duplo_yellow_on_blueduplo_yellow_on_bluecubescubesHow was this tested (or how to run locally)
tests/test_so101_block_stacking.py,tests/test_so101_duplo.py,tests/test_so101_realism.py. 47 passed in 10.6 s locally (MuJoCo 3.12.0, Python 3.13). They cover the gripper constants against the live model, fingertips never touching the table, the wrist-roll decoupling, the Duplo mesh matching LDraw 3437 and seating at exactly one body height, a 45°-rotated brick not scoring as stacked, success having to persist, and the expert on the seeds that previously failed.pytest.importorskip("mujoco"), so they skip rather than fail where MuJoCo isn't installed (e.g. the base-deps CI job).pytest -q tests/test_so101_block_stacking.py tests/test_so101_duplo.py tests/test_so101_realism.pymujoco>=3.2; dropMUJOCO_GL=eglif you have a display):First run downloads the SO-ARM100 description and the LDraw part and caches them under
$HF_LEROBOT_HOME/.Checklist (required before merge)
pre-commit run -a) — not run: pre-commit/ruff aren't installed in this environmentpytest) — the three new test files pass; the full suite was not runexamples/so101_block_stacking/README.md)Reviewer notes
main(4aaff99b), so it applies without conflicts.examples/so101_block_stacking/plus three new test files. The one library change is thelerobot-dataset-vizmulti-episode view (commit516989ce), which doesn't depend on the sim. Happy to split it into its own PR if you'd rather review it separately.expert.py(~1,070 lines) carries most of the behaviour, andpieces.pyholds the collision model the Duplo results depend on. The README's "What the gripper can actually do" section documents the arm properties that drove the design, since those fail silently rather than loudly.🤖 Generated with Claude Code