Why Robots work in Simulation but fail in Reality One of the most frustrating moments in robotics: Everything works perfectly in simulation. Then you deploy it on a real robot and suddenly: The grasp misses The arm shakes The robot drifts Contact becomes unstable The motion looks correct, but the task still fails How do you solve the sim to real problem? At first, it sounds simple. Just move the code from simulation onto hardware. But the gap between simulation and reality is much larger than most people think. Simulation environments are extremely clean. The table is flat. Object geometry is accurate. Friction is predefined. Sensors are stable. Robot joints behave exactly as expected. But the real world is messy. Lighting changes. Depth sensors drift. Objects reflect light differently. Motors have delay. Joints have backlash. Contact forces behave unpredictably. And robotics is a chain reaction. A small perception error becomes a planning error. The planning error becomes a control error. The control error becomes an execution error. Eventually, the robot misses the grasp by a few centimeters and the entire task fails. And The hardest part is usually contact. Humans think tasks like: grasping a cup, opening a door, inserting an object, pushing a box are trivial. For robots, these are extremely difficult because contact is not clean physics. A tiny shift in friction, force, or surface geometry can completely change the result. In simulation, objects are usually “well behaved.” In reality: objects slip contact points shift surfaces deform collisions happen unexpectedly This is why many robotic tasks fail not because the policy is fundamentally wrong, but because reality itself introduces uncertainty. Sensors are also less reliable than people think. The robot’s perception already contains error: camera noise unstable depth estimation occlusion pose estimation drift changing lighting conditions Sometimes the model itself is fine, but the input is already slightly wrong. By the time the error propagates to the end effector, the grasp fails. The robot hardware itself is also imperfect. Motors have latency. Controllers have frequency limits. Actuators have error. Different loads change behavior. In simulation, the robot follows commands perfectly. In reality, it may move slightly slower, slightly off target, or slightly unstable. Those tiny differences are fatal in robotics because robots physically interact with the world. Sim2Real being difficult does not mean simulation is useless. Simulation is still incredibly valuable: they are cheap, safe, scalable and reproducible. A better way to think about simulation is: Simulation is the training ground, not the final battlefield. Modern Sim2Real methods usually combine multiple approaches: making simulation more realistic, adding domain randomization, randomizing lighting, friction, object positions, and sensor noise, fine-tuning with real-world data. The goal is not to make the robot adapt to one perfect virtual world. The goal is to make the robot robust enough to survive an imperfect real one. The most important lesson in robotics is: Success in simulation is only the first step. The real test begins when the robot touches the real world. Video Credit: Kevin Zakka

We are excited to re-introduce our Behavior Foundation Model for Humanoid Robots, built upon a unified perspective of diverse WBC tasks, as a promising step toward a foundation model for general humanoid control. 🔗Website: bfm4humanoid.github.io 📜Paper: arxiv.org/abs/2509.13780

🔹Topic: A framework for controlling physically simulated football players to perform dribbling, trapping, moving, and kicking—seamlessly blending realism and interactivity.

We’re presenting AlphaProteo: an AI system for designing novel proteins that bind more successfully to target molecules. 🧬 It could help scientists better understand how biological systems function, save time in research, advance drug design and more. 🧵 dpmd.ai/3XuMqbX

I just had 150 Rainbow Points dropped into my wallet — everybody has at least 100 points waiting for them, but you might have more! Claim your drop: rainbow.me/points?ref=RB36PC

Starship launch on track for tomorrow morning

Preparing for launch on Saturday

Meta presents EMU VIDEO: Factorizing Text-to-Video Generation by Explicit Image Conditioning blog: emu-video.metademolab.com/ paper: emu-video.metademolab.com/as… present EMU VIDEO, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image. We identify critical design decisions–adjusted noise schedules for diffusion, and multi-stage training–that enable us to directly generate high quality and high resolution videos, without requiring a deep cascade of models as in prior work. In human evaluations, our generated videos are strongly preferred in quality compared to all prior work– 81% vs. Google’s Imagen Video, 90% vs. Nvidia’s PYOCO, and 96% vs. Meta’s Make-A-Video. Our model outperforms commercial solutions such as RunwayML’s Gen2 and Pika Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user’s text prompt, where our generations are preferred 96% over prior work.

New AI research paper from Meta — MART, or Multi-round Automatic Red-Teaming is a framework for improving LLM safety that trains an adversarial and target LLM through automatic iterative adversarial red-teaming. Details in the paper ➡️ bit.ly/40H1l2z

Drivable 3D Gaussian Avatars paper page: huggingface.co/papers/2311.0… present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

After learning today’s news, this is the message I sent to the OpenAI team:

i loved my time at openai. it was transformative for me personally, and hopefully the world a little bit. most of all i loved working with such talented people. will have more to say about what’s next later. 🫡

Pika 1.0, Coming Soon. Check out our 3D animation teaser made completely by text-to-video AI. 🙌

Check out @Andre3035858461 & @vivek_myers's blog post on GRIF, our method for vision-language policies with language alignment: bair.berkeley.edu/blog/2023/… Semi-supervised learning of language following policies.

Introducing Video Language Planning! video-language-planning.gith… By planning across the space of generated videos/language, we can synthesize long-horizon video plans and solve much longer horizon tasks than existing baseline (such as RT-2 and PALM-E). (1/5)

How can we leverage Stable Diffusion’s internet-scale pre-trained knowledge to enhance zero-shot robotic manipulation? Check out our new paper😉 SuSIE: Subgoal Synthesis via Image Editing rail-berkeley.github.io/susi… Detailed threads from Sergey and Aviral below:

Neural nets extrapolate unpredictably, right? It turns out that for reasonable shifts, extrapolation can be much more structured: neural nets can be made to "hedge" for OOD inputs, reverting to the "optimal constant prediction" (OCS). A thread about our recent paper below:

An OS that boots to a baby Llama 2 github.com/trholding/llama2.… Standalone, Binary Portable, Bootable I expected that my "Llama 2 inference code in a single .c file" would go places, but this really stretches the imagination :) And why not, do we really need all this stuff?