I have some mixed feelings about this result: On the one hand, it's genuinely impressive. I didn't know that Shampoo could be configured to perform this well on the benchmark. On the other hand, the way this performance boost was achieved seems difficult to call "Vanilla," for the following reason: According to @_arohan_, the boost depends upon fixing a numerical linear algebra issue that he observed to occur in my initial standard DistributedShampoo run. He fixed the issue by enabling the flag rank_deficient_stability_config=PseudoInverseConfig(). Here's the problem: This is an undocumented flag. It is contained within the 12,000-line DistributedShampoo codebase, but it does not appear in any user-facing documentation. As a result, if someone tries to train a model using DistributedShampoo without either (a) knowing about this special undocumented flag or (b) being prepared to detect and fix the numerical linear algebra issues that may occur without it, then they won't be able to achieve @_arohan_'s level of Shampoo performance. This level of effort would be considered atypical for mere hyperparameter tuning. -- [Note on Muon baseline in plot below: Rohan's post compared Shampoo to a slightly undertuned Muon baseline from 2026/05/01, which reached the target loss in 3375 steps. This resulted in a 50-step gap between Shampoo and Muon. In the figure below I'm using the up-to-date 2026/05/03 baseline, which reaches the target in 3325 steps. This results in the step-counts exactly matching between Muon and the tuned/stabilized Shampoo variant.]

June 9th Researcher Reciprocity License "if you train on it, you let us generate - reverse terms of use void" Status quo 1. We teach frontier devs with ICLR/NeurIPS papers, OSS Github contributions 2. They use it to make frontier models 3. Then ban us from exploring our ideas We need a new license, original thinkers can't be an underclass to a tyrannical researcher fiefdom

Humanity's Last Hackathon is NOW OPEN for registration. This is not a normal hackathon. You will be judged on the context, not the code! Use Codex @OpenAIDevs to build and optimize models for local inference (kernels on Max metal). Submit through @GPU_MODE. Climb the leaderboard. Top performers qualify for the final battle. Launches May 4th. Registration is live now.

AMD's @AnushElangovan explains why he thinks his company's open source ethos combined with agentic AI superpowers their leverage as a company: Because AMD publishes a lot of technical details about its hardware, when engineers use AI tools, the models already “understand” AMD’s systems and can help write code for them, debug them, or even generate new tools. And that makes developers more productive on AMD hardware without AMD having to do all the work internally. "AMD has had this ethos of open source, which really plays to our advantage. Every frontier model that I use has already seen every bit of AMD source code." "It'll rewrite my spec for me because it's already in the training data. Which you can't get from closed ecosystems." "In fact I built a virtual GPU simulator just based off our public specs, and now I'm running it on the GPU. So now I can run cross-generational GPU simulations on existing hardware." "We have that advantage. And we've run a Dev Day contest where we generated more tokens on AMD — Triton kernels and HIP kernels — than existed on the internet at the time." "So now that's all part of the pre-training data. It's a superpower because now you're open source, and you're agentically accelerating this process."

We’re sponsoring a @GPU_MODE hackathon in Paris on April 9, to conclude the PyTorch Conference Europe 🇪🇺 Our grand prize? 48 hours on GB300 NVL72. Join us with the teams behind @PyTorch, @PrimeIntellect, @SemiAnalysis_, @sestercegroup, and more! luma.com/gpu-mode-paris-2026…

Trtllmgen kernels are now open. Fastest prefill and decode kernels for our target workloads. We wrote these to win InferenceX, MLPerf, other benchmarks. Powering some of today’s top served models. Dive in, learn, use them, or level up your own. Enjoy. github.com/flashinfer-ai/fla…