Day 10/365 of LLM Inference Today's focus: Prefill and Decode Disaggregation - why the best inference systems separate them completely Yesterday we covered Pipeline Parallelism. Today is about one of the most important architectural shifts happening in production LLM serving right now. Quick recap of the problem: Back on Day 2 we established that inference has two phases. Prefill is compute-bound. Decode is memory-bandwidth-bound. They have completely different resource profiles and completely different bottlenecks. So why do we run them on the same GPU? How traditional inference servers handle this: In most inference servers today prefill and decode happen on the same set of GPUs interleaved together. A decode request and a prefill request compete for the same GPU resources at the same time. This creates a fundamental conflict. Long prefill requests stall ongoing decode requests. Latency spikes. Time to first token suffers. The GPU is never fully optimised for either workload. What disaggregation does: Separate the two phases onto dedicated hardware entirely. Prefill nodes handle only prompt processing. They are optimised for raw compute throughput. High tensor parallelism, large batch prefill, maximise FLOPS utilisation. Decode nodes handle only token generation. They are optimised for memory bandwidth. Efficient KV Cache management, continuous batching, minimise memory bottlenecks. Each pool of GPUs is tuned independently for exactly one job. Why this matters at scale: → Time to first token drops dramatically on prefill nodes → Decode throughput improves because it is never interrupted by prefill → You can scale prefill and decode capacity independently based on actual load → KV Cache can be transferred between nodes allowing smarter memory management across the cluster Who is shipping this today: This is not theoretical. Disaggregated prefill and decode is already in production. → Deepseek built their entire serving infrastructure around this model → NVIDIA TensorRT-LLM added disaggregated serving support → vLLM has active development on prefill decode disaggregation right now → Microsoft Splitwise paper formalised the architecture in 2024 The tradeoffs → KV Cache transfer between prefill and decode nodes adds network overhead → Cluster orchestration becomes significantly more complex → Requires careful load balancing to avoid prefill nodes becoming a bottleneck → Not worth the complexity at small scale, this is a large fleet problem Most teams running fewer than 8 GPUs do not need this. But if you are operating inference at scale and your time to first token is hurting, disaggregation is where the industry is heading and fast. Day 11 tomorrow: Flash Attention and how we made the attention operation itself faster 🔦 #LLMInference #PrefillDecodeDisaggregation #MachineLearning #GenAI #GPU #LLM #AIEngineering #MLOps #vLLM #Splitwise #DeepSeek

𝐉𝐨𝐢𝐧 𝐮𝐬 𝐟𝐨𝐫 𝐨𝐮𝐫 𝐀𝐩𝐫𝐢𝐥 𝐋𝐌𝐂𝐚𝐜𝐡𝐞 𝐎𝐟𝐟𝐢𝐜𝐞 𝐇𝐨𝐮𝐫 𝐭𝐡𝐢𝐬 𝐓𝐡𝐮𝐫𝐬𝐝𝐚𝐲! @ChengYihuaA, CTO of @Tensormesh, will be sharing the 𝐧𝐞𝐰 𝐌𝐏 𝐌𝐨𝐝𝐞 𝐝𝐞𝐬𝐢𝐠𝐧 for #LMCache. We’ll cover how this architecture handles model parallelism and KV caches, its performance, and what’s coming next on the roadmap. Interested in learning how this new feature unlocks 𝟏𝟎𝐱 𝐟𝐚𝐬𝐭𝐞𝐫 𝐌𝐨𝐄 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 to accelerate generation and reduce user latency? Come hear the latest updates, get a look at what’s ahead, and ask any questions you may have. 📅 Thursday, April 9, 2026 | 11:00 AM–12:00 PM PST 🔗 Join here: meet.google.com/ehe-fiap-mzc #MoE #AI #inference #LMCache #KVCache #vLLM

Day 6/365 of LLM Inference Today's focus: Speculative Decoding - how we trick the model into going faster Yesterday we covered PagedAttention. Today is about one of the most clever tricks in modern LLM inference. The core problem with decoding Decoding is sequential. One token at a time. You cannot parallelize it because each token depends on the previous one. The big model sits waiting between each step. Pure latency. The insight behind speculative decoding What if a small cheap model drafted multiple tokens ahead and the big model just verified them in parallel? That is exactly what speculative decoding does. How it works: → A small draft model generates N tokens speculatively and quickly → The large target model verifies all N tokens in a single forward pass → If the draft tokens are correct, you accept them all at once → If a draft token is wrong, you reject from that point and fall back to normal decoding The big model still does all the verification. Output quality is mathematically identical. But instead of N sequential forward passes on the large model, you get N tokens from one pass. Why this is a big deal: The large model's forward pass cost is roughly the same whether it processes 1 token or 8 tokens. Verification is nearly free compared to generation. On tasks where the draft model guesses well (code, repetitive text, structured outputs) you can see 2x to 3x latency improvements with zero quality loss. The tradeoffs: → Needs a good draft model that matches the target model's distribution → Works best on predictable output patterns → Adds system complexity with two models running together → Acceptance rate varies heavily by task Speculative decoding is now shipping in production inside vLLM, TensorRT-LLM and Hugging Face TGI. Day 7 tomorrow: Quantization and how we shrink models without destroying them 🔢 #LLMInference #SpeculativeDecoding #MachineLearning #GenAI #GPU #LLM #AIEngineering #MLOps #vLLM #LLMOptimization

Day 3/30 of LLM Inference Today's focus: KV Cache - why it exists and what it actually does Yesterday we covered prefill vs decoding. Today is about the thing that makes decoding not completely terrible. The problem without KV Cache: Every time the model generates a new token, it would need to recompute attention over every single previous token from scratch. For a 2000 token context, that's 2000 recomputations per new token. Pure waste. What KV Cache does: During prefill, the model computes Key and Value matrices for every token in your prompt. Instead of throwing them away, we store them in GPU memory. During decoding, each new token only computes its own K and V, then reuses everything already cached. No recomputation. Just a memory lookup. The tradeoff: KV Cache trades compute for memory. The cache grows with every new token generated. For long contexts and large batches this blows up fast. This is exactly why: → Context length is expensive → Batch size has limits → vLLM's PagedAttention was a big deal (Day 5) KV Cache is the single most important optimization in LLM inference today. Everything else builds on top of it. Day 4 tomorrow: Batching and throughput 📦 #LLMInference #KVCache #MachineLearning #GenAI #GPU #LLM #AIEngineering #MLOps #Transformers