Introducing our work @GoogleAI CALM: Confident Adaptive Language Modeling 🧘 Large Language Models don't need their full size for every generated token. We develop an Early Exit framework to significantly #accelerate decoding from #Transformers! 🔗: arxiv.org/abs/2207.07061 🧵1/

Gemma 4 MTP just got officially merged into llama.cpp This means you can use Gemma 4 QAT MTP for a lightweight super fast setup. Excited to see what the community builds with it github.com/ggml-org/llama.cp…

Gemma 4 12B MTP speculative decoding on mlx-vlm 🚀 We benchmarked MTP on Gemma 4 12B across all 4 modalities in mlx-vlm — and it speeds up everything: text, image, audio, and combined audio image, up to 1.72× and 80 tok/s on a single M3 Ultra. Get started today: > uv pip install -U mlx-vlm github.com/Blaizzy/mlx-vlm

Local AI at 400 tok/s, zero loss, Gemma 4 31B 🤯 Google's MTP approach is way better than Qwen's because it SCALES with MTP parameter but Qwen peaks at MTP=3. I can now have FULL BF16 Gemma 4 31B at 400 tok/s on RTX 6000 Pro using MTP=100, video coming soon!

Gemma 4 up to 3x faster, directly in your phone! 🚀 Check out the difference Speculative Decoding makes! Multi-Token Prediction (MTP) is supercharging inference speeds for Gemma 4.

I benchmarked Google’s new MTP for Gemma 4 31B using vLLM with 4 speculative tokens, a fairly conservative setup. Results: - Much higher throughput than Qwen3.6’s MTP - Lower latency too, helped by Gemma 4 generating fewer tokens - For coding tasks with reasoning enabled, Gemma 4 is now at least 6x faster than Qwen3.6. So you can generate 5 outputs, run your tests to select the best one, and it would still be cheaper than a single output by Qwen3.6. I’ve updated my full comparison with the new numbers: kaitchup.substack.com/p/qwen… I also confirmed what others have reported: Gemma 4’s MTP handles a high number of speculative tokens very well. On simple text generation, I’m now testing values above 10 and reached 129 tok/s on an RTX Pro 6000, compared with 20 tok/s without MTP. Next step: confirming how this translates to real tasks.

Google dropped MTP versions of Gemma4. Ran them on my DGX Spark. The 31B dense model went from 3.94 → 8.91 tok/s. That's 126%. Full results: [26B A4B] > 25.24 → 31.69 tok/s ( 25.6%) > TTFT 755 → 332ms (-56%) [31B] > 3.94 → 8.91 tok/s ( 126%) > TTFT 599 → 378ms (-37%) If you're not running MTP, you're leaving free perf on the table.

🚨 Google just made Gemma 4 up to 3x faster with MTP ⚡ Same quality, way more speed. It predicts multiple tokens at once and verifies them in parallel, removing latency bottlenecks. You can also run powerful models locally on mobile like me using Google AI Edge Gallery.

🚀 Day-0 MTP support for Gemma4 now available at vLLM with ready-to-use docker image! ⚡️Enjoy up to 3x faster decoding performance to supercharge your development with zero quality degradation! Check out the full vLLM recipes for Gemma 4 model series👇 recipes.vllm.ai/Google/gemma…

2x speedup on Mac M5 is real.

Gemma 4 just got even faster! We're releasing Multi-Token Prediction (MTP) drafters that deliver up to a 3x speedup, without any degradation in output quality or reasoning logic.

Gemma v4 MTP drafters are out! 🚀 I was quite involved with the development of this, so I thought I might as well do a short summary of the key bits. :) tl;dr: Nothing reduces your decode time like drafting. ⚡️ 🔗 blog.google/innovation-and-a…

Si sois amantes de la familia de modelos Gemma 4, ojito a esto que acaban de meterle un boost de rendimiento que puede llegar hasta x3 más rápido, según el modelo, sin pérdida de calidad en sus respuestas o razonamientos 🔥

Gemma 4: Now up to 3x Faster. ⚡ Same quality, way more speed. Our new MTP drafters allow Gemma 4 to predict multiple tokens at once, effectively tripling your output speed without compromising intelligence.

And checkout @MaartenGr really cool visual guide to Gemma 4 that now also includes a detailed visual explanation of the drafter: newsletter.maartengrootendor…

See more details at blog.google/innovation-and-a…

Better model for about the same speed. More to come!