OpenVLA–LIBERO Spatial–Isaac Sim Benchmark

Documentation and visual assets for transferring the LIBERO Spatial tasks to Isaac Sim 6.0.1 with a Franka robot and an OpenVLA server.

한국어 보고서는 docs/benchmark_report_ko.md, 영어 보고서는 docs/benchmark_report_en.md에서 확인할 수 있다.

Result at a glance

Item	Value
Tasks	0–9
Episodes	500 (50 per task)
Seed	7
Successes	43
Success rate	8.6%
Runtime errors	0
Maximum policy steps	220
IK-failure episodes	311/500 (62.2%)
Effective server crop	0.35, 224×224

Repository layout

.
├── README.md
├── .gitignore
├── bridge/
│   ├── libero_spatial_environment.yaml
│   ├── libero_spatial_env_isaac601.py
│   ├── run_openvla_isaac_bridge.py
│   ├── adapters/
│   ├── contracts/
│   └── openvla_server.py
├── configs/
│   ├── libero_spatial_task_poses.json
│   └── README.md
├── docs/
│   ├── benchmark_report_ko.md
│   ├── benchmark_report_en.md
│   ├── architecture.md
│   ├── reproducibility.md
│   ├── action_observation_contract.md
│   ├── success_criteria.md
│   ├── failure_analysis.md
│   ├── data_format.md
│   ├── benchmark_protocol.md
│   └── visuals.md
├── figures/
├── results/
├── assets/
│   ├── fixed_objects.usd        # optional project-specific asset
│   └── README.md
└── scripts/
    ├── make_figures.py
    └── assemble_github_repo.sh

Documentation

Document	Purpose
benchmark_report_ko.md	한국어 종합 보고서
benchmark_report_en.md	English benchmark report
architecture.md	System architecture and control flow
reproducibility.md	Environment, commands, and output paths
action_observation_contract.md	RGB, depth, 7D action, frames, and gripper conventions
success_criteria.md	Strict placement/release success definition
failure_analysis.md	Task-wise failure and IK analysis
data_format.md	JSON, logs, video, and episode artifact layout
benchmark_protocol.md	Fixed variables and benchmark procedure
visuals.md	Figure index and Markdown embedding examples

Important scope notes

• The benchmark uses one fixed libero_agentview RGB camera for OpenVLA.
• Depth is recorded for diagnostics; it is not used as an OpenVLA input in this run.
• The original LIBERO instruction is preserved. No scene graph is added to the policy prompt.
• Arm actions are converted through the canonical action boundary and Lula IK.
• Initialization and finger-only gripper commands use direct joint paths by design.
• The evaluator reads actual object pose, support, footprint, release, and clearance state.
• There is no bowl-coordinate teleportation or forced-success flag in the documented evaluator path.

Regenerating the figures

python3 scripts/make_figures.py \
  --benchmark-json data/benchmark_report.json \
  --summary-json data/summary.json \
  --output-dir figures

The figures use the latest Isaac Sim benchmark artifacts: 43/500 successes. Do not mix them with older reports that contain the separate 422/500 result.

Assembling the runnable GitHub tree

After extracting this package, rename the local runner to run_openvla_isaac_bridge.py and run:

bash scripts/assemble_github_repo.sh /path/to/Libero-OpenVLA-Benchmark-in-IsaacSim

The script uses mkdir and cp to collect the bridge source, adapter and contract files, pose JSON, optional custom USD, benchmark summaries, and server health/startup logs from the paths documented in docs/reproducibility.md.

This package contains documentation, aggregate result data, figure sources, and visual assets. It does not bundle Isaac Sim, model weights, credentials, or large USD assets; place the working bridge source under bridge/ and document external asset acquisition separately.