OmniSim

The simulator you can talk to. Describe a scene and an agent builds it; describe a behaviour and
an agent wires the controller. Everything runs over plain HTTP/JSON, so the agent loads the world,
steps physics, takes a screenshot and hot-reloads — without leaving the conversation.

Built by agents, for agents: the HTTP harness, the Newton physics integration, the cloth and
soft-body stack, the RL pipeline and the ROS 2 sidecar were written by an AI agent under human
direction. Physics is Newton, the only backend.
Apache-2.0, with a first-party MCP server for Claude Code and Cursor,
and a ROS 2 sidecar speaking the simulation_interfaces standard.

Join the public beta · Builder challenge · For research labs · Latest release ·
Demos · Agent entry point · Protocol

The OmniArm 6 Universal Pick demo uses a top-down depth camera to choose grasp
points on arbitrary, previously unmodelled shapes and move them from the bin to
the tote. Normal picks use no object registry, classifier or authored pick
anchors; the controller documents its limited recovery and suction-bookkeeping
paths. Play the MP4.

Public beta: we are looking for the first ten external developers willing
to spend 20 minutes installing OmniSim, running one demo, and reporting the
first confusing or broken step. Windows has the first downloadable package;
Linux is a source build; macOS is not supported.
Take the 20-minute challenge →

Run a real robot demo

Three minutes on Windows. About half an hour on Linux, because you build it.

1. Get OmniSim.

   • Windows 10/11 — install the asset from the
     latest release
     (~600 MB). This is the only prebuilt package.
   • Linux — Ubuntu 24.04, built from source:
     bash scripts/install/linux_bootstrap.sh. Budget 25–45 minutes; most of it
     is the compile. Details: quickstart.
     ⚠ Ubuntu 22.04 does not work — its Python 3.10 makes newton raise at
     ModelBuilder(), so worlds load and nothing moves.
   • macOS — not supported. There is no package, no verified build, and
     Newton physics is unverified. Use Windows or Ubuntu 24.04.

2. Check the install. Open a terminal in the OmniSim directory:

   python -m omnisim doctor
   

   It ends on a VERDICT line, and exits non-zero if the install cannot run —
   most usefully when the Newton runtime is absent, which is not a degraded
   mode but an install where nothing ever falls. On Windows without Python,
   run omnisim.bat doctor instead: it uses the interpreter in the package.

3. See a robot move.

   python -m omnisim demo
   

   That is the real friction-grasp demo above. python -m omnisim demos lists
   all 50, by category.

Then choose the shortest route to your own work:

• Coding agent or MCP client: read AGENTS.md. Opening this
  directory in Claude Code registers the first-party
  OmniSim MCP server automatically — the checked-in
  .mcp.json does it, with no install. Start the harness it proxies with
  python -m omnisim harness. There is also a packaged
  OmniSim Codex plugin.
• ROS 2: start with the verified
  ros2_control Husky diff-drive example.
• Your robot or research task: use the URDF/STEP workflows,
  or ask OmniLink to port one bounded, measurable pilot.

If any step is confusing or fails, that is exactly what the
public beta is designed to capture.

How OmniSim compares

Every OmniSim cell is measured by us and is reproducible from this repo. Every row names the
machines that produced it and we never average across them: the §3 throughput rows are three
machines, the §3 workload rows and the capability matrix are two, and a row that is still one
machine says so in the row. The machines are a laptop RTX 3060 (6 GB), Ryzen 16-thread, Windows
11 (9722d23d12a3), an RTX A4500 / AMD EPYC 7352, Ubuntu 22.04 (8ab788c4c833), and RunPod
RTX 4000 Ada (c72ce5632c81) and RTX 4090 (b5dadd645b1f). Competitor cells are their own
published documentation, dated and linked; we did not measure their engines. Where we lose is in
its own section.

1. Agent-native

2. Performance and resources

3. What that hardware actually runs

Every row says which machines produced it, listed separately and never averaged. Throughput:
three machines (campaign, 2026-08-17):

Everything else, with its coverage stated per row
(campaign, 2026-08-17):

4. Beyond rigid bodies — cloth, soft bodies, cables

The grasp numbers behind that first row — tracking error −1.50 mm on a 616-particle T-shirt
against −173.06 mm for a negative control whose jaws never close, corroborated by a second
instrument, plus the hem-edge target that misses and is reported as a miss — are in
cloth-simulation.md, with the three disclosures that travel
with them: the fabric is pinned, so these are tracking and not load-bearing numbers; self-contact
must be off to grasp and on to drape; and the composed fold is not demonstrated.

5. Newton maturity

6. Licence

Caveats, sources and the claim-by-claim comparison with every entry marked verified / unaudited /
vendor-claim: docs/developer/simulator-comparison.md.

What OmniSim is worse at

• ROS 2 support is new and incomplete. OmniSim implements the ROS 2
  simulation_interfaces standard plus /clock, /tf, JointState,
  /odom, cmd_vel, sensor topics (Imu, LaserScan, GPS) and a ros2_control
SystemInterface — but that last one is verified for velocity-commanded bases only
  (diff_drive_controller on the Husky). MoveIt is still out of reach, because OmniSim's arm
  bridge treats a joint command as a goal and answers 409 busy to a setpoint arriving while the
  previous one is still interpolating — a trajectory would land in pieces. Nav2 has never been
  brought up against OmniSim, and OmniSim is not in the ros2_control simulator registry. Sensor
  coverage is partial by measurement, not by omission: OmniSim's Gyro and Accelerometer produce
  no usable data, so Imu ships a real orientation and declares those two components absent, and no
  robot in the tree has a camera. For a lab whose stack is ROS 2 end to end, Gazebo remains better
  integrated.
• Not photoreal. Isaac Sim's RTX renderer is a different class. Ours is wgpu-native (Vulkan / D3D12 / Metal): a real-time raster stack whose global illumination is baked — an off-frame path trace into an irradiance probe volume, not per-frame ray tracing.
• Sim-to-real is unproven — zero physical-robot transfer.
• No free-standing humanoid walk — every G1 result uses a weight-bearing balance harness.
  Quadrupeds carry none.
• Newton is young and we removed our fallback. macOS is untested, so it has no verified physics.
• Single-GPU training only. No multi-GPU or multi-node; Isaac Lab publishes a 16-GPU ladder.
  Our figures are one GPU, and every number above says which one.
• 78% of measured capabilities work (45 probes, run on two machines — 43 of 45 verdicts
  agree, and both disagreements are attributed in the
  campaign, neither to the hardware).
  Restitution is unimplemented, and runtime scene mutation is non-physical in both directions:
  deleted nodes keep colliding, and spawned nodes are never registered with the solver at all.
  Compound colliders drop all but the first child by default. A motorised BallJoint still does not
  actuate (its Hinge2Joint sibling now does).
  Capability matrix · CHANGELOG.
• A mujoco_warp scene silently drops constraint rows past 256. Measured on both machines:
  16 driven four-wheeled rovers peak at 336 and 328 rows against a cap that does not grow, with a
  clean log and exit code 0. Raise WorldInfo.newtonNjmax before you build a fleet.
• Smaller ecosystem than Gazebo — fewer robots, fewer worlds, no package index.

Get started

The three-minute path above is the fastest start. Contributors who want a
source build can clone it, open the folder in a coding agent, and ask:

Set up OmniSim from this fresh clone — install whatever the toolchain needs, build it,
and then launch the warehouse Husky demo so I can watch it.

The agent follows AGENTS.md (auto-loaded) and the
quickstart. First build is 5–25 minutes. By hand instead:
build_omni.bat on Windows, bash scripts/install/linux_bootstrap.sh on Linux.

Then run python -m omnisim doctor — it reports the ground truth about your clone. ⚠️ Newton is
the only backend, so a build without its runtime has no physics at all and otherwise fails
quietly.

What you can ask for

"Launch the warehouse demo."
"Generate a Mars world with a 5-Husky fleet and run it headless for 30 seconds."
"Add three red cylinders in front of the Husky and make it avoid them."
"Wire a Jackal on a flat platform, expose it on HTTP, and drive it forward 2 m."
"Why isn't the camera seeing the red cylinder?"

The agent builds the world, edits controllers, runs the simulator, and verifies its own work through
the validation harness. For runtime control of robots in a live scene, point
OmniLink agents at the per-robot HTTP bridges
(protocol).

Robots

Per-robot status is in DEMOS.md. One caveat on OmniSim's own reference robots:
OmniTug 500 is a visual-only prop with no collider, so it is positioned kinematically and does
not participate in contact. OmniArm 6 and OmniArm 7 are fully simulated, and OmniArm 6 is
verified holding pose under gravity.

Teaching robots to move

Legged policies are made by Shadowing — train against a reference trajectory verified feasible
before learning starts, in-engine, so train and deploy share one solver. Fifteen skills ship across
G1, H1, Go2 and OmniQuad; BATON composes them into sequences.

python -m omnisim policy list                  # the catalogue
python -m omnisim policy sequence box_delivery # run a BATON demo

⚠️ Our humanoid demos run on a weight-bearing balance harness and are not free-standing walks.
Unitree's own policies re-hosted unchanged in OmniSim do walk free-standing, so the engine carries
it — training a policy that does is the open problem.

Shadowing · Skill Library ·
canonical RL status

Docs

Licence · Brand · Bugs · Funding

Apache 2.0 (LICENSE · NOTICE). The OmniSim name and orb mark are OmniLink
trademarks (TRADEMARKS.md) — open code, protected brand: fork and ship, but rename a
modified fork. Bugs go to GitHub Issues; sponsorship funds the engine work
(Sponsor · SPONSORS.md).

Attribution

Built on Webots (Cyberbotics Ltd., open-sourced under
Apache 2.0 in 2018). Upstream copyright on derived files is preserved; files retaining Cyberbotics
headers remain © Cyberbotics. OmniSim is an independent fork, not affiliated with or endorsed by
Cyberbotics. See NOTICE for the full derivation.