Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Attent…

Initial runs at stage 2 of explain our finding: that QKV in deep attention layers need drastically different inputs. The first experiment: can we learn an attention residual for QKV individually, to see which of the previous layers each of them needs the most? Two findings: (1) Deep value vectors do not attend to recent layers, and are fixated on the initial token embedding, as is found by our paper. (2) The surprise: Deep value vectors also have a large attention on almost every MLP layer, and skip almost every attention layer. More experiments on the way. Paper: arxiv.org/abs/2606.02780

"If you are not working 7 days per week, you are going to lose". Corgi Insurance is the most intense workplace culture in startups. - The company works 7 days per week. - Founder (@nico_laqua) lives and sleeps in the office. - He built a cafe in the office because there was no local cafe that was open 24/7. - 2/3 of the first 30 team members have the Corgi logo as a tattoo. Today I went behind the scenes with Nico, who has used this culture to scale the company to a $2.6BN valuation in just two years. My condensed notes below: 1. If You Are Not Working 7 Days Per Week, You Are Going to Lose: Whatever you can get done in 5 days, you'll get more done in 6 and 7. If you are trying to solve the world’s hardest problems, a standard 5-day workweek will not cut it. 2. Work Trials Repel the Mediocre: Corgi forces candidates into mock work trials over the weekend. If seeing a full office on a Saturday scares them, they don't belong. True intensity acts as a natural filter to attract killers and repel clock-watchers. 3. Lead from the Front Lines You can’t demand 7-day weeks while sitting on a yacht. Nico sleeps 3–4 hours a night on a mattress inside the office. If you want your troops to bleed, you have to be in the trenches with them. 4. Culture Only Means One Thing: Winning Forget superficial jargon like "hackers" or "ex-founders." Strip away the corporate fluff. A great startup culture is aggressively optimized around one single word: Winning. 5. Lifespan vs. Victories Building something world-historic requires radical sacrifice. When asked if he'd rather build a trillion-dollar company and die at 50, or fail and live to 80, the answer was easy. "I would rather measure my lifespan in victories." 6. Reject the Comfort of "Quiet Quitting." If you are operating in a hyper-growth environment and your days off happen to be Saturday and Sunday every single week, you are quiet quitting. To win, you must deliberately bypass the off-ramps of personal comfort and low volatility. Corgi isn't for everyone—and that’s exactly the point.

Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper judgment, more honesty about its own progress, and the ability to work independently for longer than its predecessors. Available today at the same price.

TPUs.

my boss: did you finish the [super duper secret] kernel i asked for last month me: i have to sleep bro