Real-world data as the foundation for robotics simulations

• We release 1,000 simulation environments for five bimanual tasks from the ABC dataset, created real-to-sim from real robot trajectories.
• Our environments show strong sim-to-real performance correlation on ABC's learned policies and reproduce real-world failures in sim.
• Policies trained in our simulation show strong sim-to-real performance on a new robot cell, achieving 4x higher success rates than policies trained on real data from other cells.

Simulation is only useful if its lessons transfer to the real world. At Versor, we are building real-to-sim-to-real systems that use real-world interactions to bridge the sim-to-real gap, turning real-world experience into realistic environments.

Central to our tech is visual system identification, which tracks objects and their articulation throughout a real world episode, and infers the physical properties of the objects, robot, and scene using the recorded interaction. Through physics stress tests and replay of real-world actions, we ensure that the environments are ready for reinforcement learning, data generation, and sim-to-real policy training.

Today, we are releasing 1,000 simulation environments reconstructed from real-world demonstrations for five manipulation tasks in the ABC dataset (Allshire et al., 2026). Every environment is a unique combination of robot cell, objects in the scene and layouts reconstructed from real trajectories.

Download Get all 1,000 simulation environments on Hugging Face ↗

Simulation environments ready for robot learning

The release spans five manipulation tasks that represent some of today's main challenges in robot learning: fine-grained manipulation (sorting screws and nuts), bimanual coordination (loading plates into a dish rack), deformable objects (placing snacks into a paper bag), and reorienting and placing objects (flipping mugs, placing bottles in a bin). Each environment was automatically reconstructed from a real-world demonstration to reproduce its scene, objects, and interactions.

Explore different layouts for each task. Hover over a real or simulated frame to magnify the same region in both.

Flip mugs upright

Real

Sim

Real

Sim

Sort screws and nuts

Real

Sim

Real

Sim

Place snacks into a paper bag

Real

Sim

Real

Sim

Three of the five tasks we release. Each pair is a frame from a real demonstration beside its reconstructed environment. Real images from the ABC dataset; reconstructed simulations by Versor.

Reconstruction lets us match the distribution of scenes that a policy encounters in the real world. Other simulation datasets sample object layouts that are significantly simpler than real-world, while our episode-centric reconstructions preserve the clutter the robot actually encounters.

Real

Versor

ABC

Beyond the five released tasks, we have also reconstructed tasks such as folding a paper box, opening a wallet, placing AirPods in their case, and unlocking a padlock, covering deformation, articulation, and precise contact. The explorer below shows 10 tasks, three from the release and seven beyond it: pick one to watch the real demonstration beside its reconstruction.

Sim-to-real policy evaluation

Simulation lets us compare checkpoints and inspect behaviors before running a policy on a physical robot, provided that performance in simulation reflects performance in the real world. We evaluate ABC’s released checkpoints in simulated reconstructions of its real-world evaluation scenes for mug flipping, bottle placement, and dishrack loading, and compare their task progress with the real robot’s.

Simulation ranks the checkpoints almost exactly as the real robot does, matching 15 of the 18 pairwise orderings. Across the four checkpoints, simulated and real task progress are strongly correlated, with Pearson correlations of 0.996 for bottle placement, 0.92 for mug flipping and 0.71 for dishrack loading.

Simulation reproduces real-world failures

Aggregate metrics help select checkpoints for deployment but policy behavior can reveal policy weaknesses, helping target new training data. Typically, while sim and real might agree in aggregates, exact behaviors do not match due to distribution shifts. With our reconstructed scenes, we observe a strong sim and real behavior correspondence, beyond aggregate metrics.

Below are three examples where we observe the same failure mode in sim and real: a mug dropped during handover, a plate slipping during pick, and a bottle missing the bin. These shared behaviors show how simulation can help identify potential failures and guide data generation and retraining for more reliable policies.

Sim-to-real transfer in a new robot cell

Beyond evaluating policies, our environments can generate the data to train them. Adapting a policy to a new robot cell usually requires collecting teleoperated demonstrations in that target cell, which can be time consuming and slows down deployments. Instead, we reconstruct a new YAM cell in simulation and fine-tune an open-source VLA only on data generated in that simulation, without collecting any real-world demonstrations in the new cell.

We test this on the mug flipping task, included in this release. The robot picks up mugs from a tray, hands them over to the other arm, flips, and places them. We start from VLA-200k, a checkpoint ABC trained on its real-world data, and compare two ways to fine-tune it: using ABC's real-world mug-flipping episodes, or using demonstrations generated in our simulation of the new cell. Neither dataset includes real-world data from the new cell. First, we show how the policies performed on the real robot; then we explain how we generated the simulation data.

Real-robot results

We evaluate each policy in the real cell across six layouts, and four trials per layout. Each evaluation has two mugs, and task progress scores an episode 0%, 50% or 100% for zero, one or two mugs left upright. Strict success 1.0 when both mugs are placed upright, and 0.0 otherwise. Without fine-tuning, VLA-200k makes little progress on the new cell. Fine-tuning on ABC's real-world data reaches 35% mean task progress and 8% strict success. Fine-tuning on Versor's simulation data reaches 60% task progress and 33% strict success (closely matching simulation performance of 59.7% task progress and 27% strict success).

None of these policies saw real-world demonstrations from the new cell, but the policy fine-tuned in simulation still achieved 33% strict success. There is still a lot of room for improvement, but this is an early signal on how simulation can supply cell-specific training data without collecting new teleoperated demonstrations.

Generating the simulation data

The strongest results came from training on simulated data of the new cell. To generate this data, we combine motion retargeting, which adapts real-world demonstrations recorded in other cells to the new cell, with reinforcement learning, where policies practice the task in simulation and learn from their attempts. Together, these methods produce perfect demonstrations, new strategies, as well as, failures and recoveries, such as setting a toppled mug upright or retrying after a missed grasp. In total, we generate 3,252 episodes, about 18 hours of robot data. Each episode has two mugs in randomized positions, with randomized lighting, camera pose, and wall and table tint.

What's next

Building our simulations from real-world data allows us to precisely capture the robot and scene properties that matter in the real world. That realism lets us train and evaluate policies, from demonstrations or reinforcement learning, and transfer to the real world. Next, as we get more real world evaluation data, we are feeding this back to the simulator, and training more reliable and performant policies, continuously improving the real-to-sim-to-real loop.

In addition, we are extending our environments to longer, contact-rich commercially valuable tasks, and to a wide range of objects and materials. To learn more about our work and help push real-to-sim-to-real forward, get in touch at founders@versor.tech.

Citation

If you use these scenes, please cite our post and the ABC paper:

@misc{versor2026abcsim,  title        = {Real-world data as the foundation for robotics simulations},  author       = {{Versor}},  year         = {2026},  howpublished = {\url{https://versor.tech/news/abc-sim/}}}@article{allshire2026abc,  title   = {Scalable Behavior Cloning with Open Data, Training, and Evaluation},  author  = {Arthur Allshire and Himanshu Gaurav Singh and Ritvik Singh and Adam Rashid and Hongsuk Choi and David McAllister and Justin Yu and Yiyuan Chen and Huang Huang and Pieter Abbeel and Xi Chen and Rocky Duan and Phillip Isola and Jitendra Malik and Fred Shentu and Guanya Shi and Philipp Wu and Angjoo Kanazawa},  journal = {arXiv preprint arXiv:2606.27375},}