What if you could not only watch a generated video, but explore it too? 🌐 Genie 3 is our groundbreaking world model that creates interactive, playable environments from a single text prompt. From photorealistic landscapes to fantasy realms, the possibilities are endless. 🧵

RobotMDM, by Disney Research, combines diffusion-based motion generation with RL to produce physics-aware humanoid motions from text prompts. Trained on human motion data with a reward surrogate for physical feasibility, it ensures realistic motions. la.disneyresearch.com/wp-con…

Segment Anything 2 (SAM 2) by @AIatMeta running 100% on-device powered by Apple CoreML! ⚡ Takes fraction of a second to run inference on Mac or iPhone! > Apache licensed optimised model checkpoints - tiny, small, base ad large! > Open source application to annote any image in a sub-second > Conversion guides for SAM2 fine tunes like Medical SAM and much more > Video support on its way P.S. We ship packaged app for running SAM2 directly on your Mac, try them out today! Links in the first comment 👇

Introducing GRID Enterprise A private GRID experience that is scalable, customizable, and seamlessly integrated into your dev pipeline. 🔗: scaledfoundations.ai/product… 🧵(1/5)

🔥Spatial intelligence needs fast, *interactive* 3D world generation 🎮 — introducing WonderWorld: generating 3D scenes interactively following your movement and content requests, and see them in <10 seconds! 🧵1/6 Web: kovenyu.com/WonderWorld/ arXiv: arxiv.org/pdf/2406.09394

can two ai agents be in love? we placed romeo and juliet in minecraft. romeo refused to eat alone. juliet shared dreams about their future. they agreed to get married.. and romeo almost got cold feet.

NeuflowV2 - Optical Flow (ONNX) "Compared to other state-of-the-art methods, our model achieves a 10x-70x speedup" Added example with Object Detection to mask the detected objects' flow. Code:github.com/ibaiGorordo/ONNX-… Video:youtu.be/S0RnlEHGNrc Neuflow: github.com/neufieldrobotics/…

UNet 3 : a U-shape encoder-decoder architecture built upon the foundation of its predecessors, i.e., UNet and UNet . It aims to capture both fine-grained details and coarse-grained semantics from full scales. idiotdeveloper.com/unet-3-pl…

2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster

1/ Our PALO approach learns new robot manipulation skills from as few as five demonstrations! The key insight is that we can use a VLM (GPT-4o) to search for the best of several semantically-equivalent language subtask decompositions for the given demonstrations. #corl2024

How can we generate high-fidelity, complex 3D scenes? @QTDSMQ's LT3SD decomposes 3D scenes into latent tree representations, with diffusion on the latent trees enabling seamless infinite 3D scene synthesis! w/ @craigleili, @MattNiessner quan-meng.github.io/projects…

Hello, world! We are World Labs, a spatial intelligence company building Large World Models (LWMs) to perceive, generate, and interact with the 3D world. Read more: worldlabs.ai/about

GameGen-o from Tencent with the first diffusion transformer model tailored for the generation of open-world video games It simulates: - character movement - dynamic environments - complex actions Looks like a gamechanger

Discover your inner Tony Stark Introducing compute.hyper.space - a new type of deep dive agentic interface. Powered by open source models from @AIatMeta @MistralAI; running on @GroqInc, or on your laptop or the @HyperspaceAI peer-to-peer AI network. No filters. No bias. No subscription fees. You are in control. cc @SmokeAwayyy @karpathy @pmarca @arthurmensch @MatthewBerman

I am asking once again — why does China have the best AI video models on the market? Is it compute? Is it data? Is it algorithmic prowess? I can get firefly video being a laggard but aren’t most US based models training on “publicly available data”?

So excited to announce @HeyGen_Official's new Avatar 3.0! Our avatars have evolved beyond lip-syncing to feature full-body dynamic motion. For the first time, our avatars' facial expressions and voice tones are dynamically generated to perfectly match the script. Available at heygen.\com today!

The western saloon demo that we built to show to folks at last GDC exhibits several aspects of Multi-Agent simulation: Characters navigate the environment. Characters can carry props from place to place. Characters can proactively initiate conversation or interaction with other characters. The AI Director organizes subgroups of characters into casts, focused on shared activities. Characters have configurable and dynamic affinities toward other characters. Characters can hear nearby conversations, and chime in if they have something relevant to add. Conversations can involve 3 characters, including human players. If you’d like to delve into these topics, take a look at our Substack, and sign up for our Tech Preview.  Links in the comments below.

OpenAI Strawberry (o1) is out! We are finally seeing the paradigm of inference-time scaling popularized and deployed in production. As Sutton said in the Bitter Lesson, there're only 2 techniques that scale indefinitely with compute: learning & search. It's time to shift focus to the latter. 1. You don't need a huge model to perform reasoning. Lots of parameters are dedicated to memorizing facts, in order to perform well in benchmarks like trivia QA. It is possible to factor out reasoning from knowledge, i.e. a small "reasoning core" that knows how to call tools like browser and code verifier. Pre-training compute may be decreased. 2. A huge amount of compute is shifted to serving inference instead of pre/post-training. LLMs are text-based simulators. By rolling out many possible strategies and scenarios in the simulator, the model will eventually converge to good solutions. The process is a well-studied problem like AlphaGo's monte carlo tree search (MCTS). 3. OpenAI must have figured out the inference scaling law a long time ago, which academia is just recently discovering. Two papers came out on Arxiv a week apart last month: - Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. Brown et al. finds that DeepSeek-Coder increases from 15.9% with one sample to 56% with 250 samples on SWE-Bench, beating Sonnet-3.5. - Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Snell et al. finds that PaLM 2-S beats a 14x larger model on MATH with test-time search. 4. Productionizing o1 is much harder than nailing the academic benchmarks. For reasoning problems in the wild, how to decide when to stop searching? What's the reward function? Success criterion? When to call tools like code interpreter in the loop? How to factor in the compute cost of those CPU processes? Their research post didn't share much. 5. Strawberry easily becomes a data flywheel. If the answer is correct, the entire search trace becomes a mini dataset of training examples, which contain both positive and negative rewards. This in turn improves the reasoning core for future versions of GPT, similar to how AlphaGo’s value network — used to evaluate quality of each board position — improves as MCTS generates more and more refined training data.

This is probably the most interesting prompting technique of 2024 🤯 Self-Harmonized Chain of Thought (ECHO) = CoT reasoning with a self-learning, adaptive and iterative refinement process ✨ 1/ ECHO begins by clustering a given dataset of questions based on their semantic similarity 2/ Each cluster then has a representative question selected and the model generates a reasoning chain for that question using zero-shot Chain of Thought (CoT) prompting - breaking down the solution into intermediate steps. 3/ During each iteration, one reasoning chain is randomly chosen for regeneration, while the remaining chains from other clusters are used as in-context examples to guide improvement. So what’s so special about this? > Reasoning patterns can cross-pollinate - as in, if one chain contains errors or knowledge gaps, other chains can help fill in those weaknesses > Reasoning chains can be regenerated and improved multiple times - leading to a well-harmonized set of solutions where errors and knowledge gaps are gradually eliminated This is like a more dynamic and scalable alternative to Google Deepmind’s "Self Discover" prompting technique but for CoT reasoning chains that adapt and improve over time across complex problem spaces. > Ziqi Jin & Wei Lu (Sept 2024). "Self-Harmonized Chain of Thought" For more on this, you can find a link to the paper and Github down below 👇