I've been saying for years that the biggest challenge for simulation in robotics is not actually the physics engine (although you do have to get that right). The real challenge is capturing the *diversity* of the real world. There was no doubt that generative AI had the potential to change that, but it's still amazing to see it take shape. Watching Nick's incredibly fast progress has convinced me that content generation might not actually be a bottleneck anymore. This is a beautiful combination of hardened tools for e.g. low-level mesh processing with the latest tools for generative asset creation, wrapped in a powerful agentic workflow. Please do give it a try and share your feedback.

Introducing Large Video Planner (LVP-14B) — a robot foundation model that actually generalizes. LVP is built on video gen, not VLA. As my final work at @MIT, LVP has all its eval tasks proposed by third parties as a maximum stress test, but it excels!🤗 boyuan.space/large-video-pla…

Boston Dynamics and Toyota Research Institute are advancing humanoid robotics with Large Behavior Models (LBMs) for Atlas.

Very proud of Nicholas, who recently shared scalable-real2sim.github.io/ (for physics-quality assets from a small amount of interaction with a robot) and is now following up with his work on scene-level generation.

Side note: I'm proud of the title of this paper, which we intentionally made pretty narrow/specific. I think that some of the most important work that we have to do as a field right now is careful empirical work to interrogate the properties of these models that we're creating.

One of the most interesting take-aways for me is that "high-performing policies need to know whether they are executing in sim or in real." A number of implications flow from that, including that sim real cotraining can decrease performance if the visual gap is too small.

New Paper: "Scalable Real2Sim: Physics-Aware Asset Generation via Robotic Pick-and-Place Setups"! 🤖 We introduce a fully automated pipeline that generates simulation-ready assets for real-world objects—no manual intervention needed! 🌐 Website: scalable-real2sim.github.io/ (1/5)⬇️

Announcing Diffusion Forcing Transformer (DFoT), our new video diffusion algorithm that generates ultra-long videos of 800 frames. DFoT enables History Guidance, a simple add-on to any existing video diffusion models for a quality boost. Website: boyuan.space/history-guidanc… (1/7)

I'm super excited to start a great new collaboration with the fantastic team at Boston Dynamics. Scott Kuindersma and I chatted with Evan Ackerman about it earlier today. spectrum.ieee.org/boston-dyn…

This #RSS2024 on July 19, we are organizing a tutorial on supervised policy learning for real world robots! Talks by @notmahi & @RussTedrake will cover the fundamentals of imitation, recent algorithms, walk-through code, and practical considerations. supervised-robot-learning.gi…

Introducing Diffusion Forcing, which unifies next-token prediction (eg LLMs) and full-seq. diffusion (eg SORA)! It offers improved performance & new sampling strategies in vision and robotics, such as stable, infinite video generation, better diffusion planning, and more! (1/8)

And if you’re at all interested in humanoids you need to check out Punyo, from TRI, a soft-body humanoid capable of whole-body manipulation. A soft robot like this could be extremely helpful for in-home robots. Check out their medium post: medium.com/toyotaresearch/me…

a bit surprised TRIs new robot has gotten little love on X current humanoid approaches seem very hand centric but we use tactile sensing across our whole body to do tasks

Check out @chichengcc's step-by-step tutorial on building the UMI gripper. We really hope to see more UMIs running in the wild. 😊