AutoLab benchmarks the actual question of 2026 frontier model agents: can they sustain long-horizon closed-loop optimization. 36 expert-curated tasks across system optimization, puzzle/challenge, model development, and CUDA kernel optimization. The dominant predictor of success is NOT initial-attempt quality, it is persistence in repeatedly benchmarking, editing, incorporating empirical feedback. Most "agentic AI" demos are competent at the first iteration and incompetent at iteration 50, which is exactly where real engineering work lives. AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? Paper: arxiv.org/abs/2606.05080

Outstanding paper on long-horizon agents. (bookmark it) Similar to humans, how do you make agents persist on a difficult task, and how is that useful? And which models today work well on this? This new work, AutoLab, explores this question and how encoding persistence in agents is beneficial for tasks such as auto research and engineering tasks. Can a model keep improving an artifact for hours, under a strict wall-clock budget, the way real research and engineering actually work? Results: AutoLab hands agents 36 expert-curated tasks across system optimization, model development, CUDA kernels, and puzzles, each starting from a correct but deliberately suboptimal baseline. Across 17 frontier models, the dominant predictor of success was not the quality of the first attempt. It was persistence, repeatedly benchmarking, editing, and folding in empirical feedback. It appears that Claude-opus-4.6 sustained that loop well. Most of the other models quit early or burned the budget, making almost no progress. Paper: arxiv.org/abs/2606.05080 Learn to build effective AI agents in our academy: academy.dair.ai/

With deep research revolutionizing research/data analysis, why are we still stuck in manually crafting data viz? Meet CoDA (huggingface.co/papers/2510.0…): The ultimate multi-agent LLM powerhouse for auto-generating stunning plots from NL queries! Handles complex data, self-refines for perfection, & smashes baselines by 41.5%🚀 Key Features: 🌟Specialized agents for metadata analysis, planning, code gen/debug, & reflection 🌟Bypasses LLM input length limits w/ metadata focus 🌟Iterative loops for robust, human-like quality checks 🌟SOTA on MatplotBench & Qwen & DA-Code #DataViz #AgenticAI #MultiAgent

Agent will take over human science!

[Accidentally deleted this earlier, reposting] 😭 #AutoLab #autoresearch We've been asking ourselves a question: if AI agents can now run hundreds of experiments overnight, how do we know whether they're actually contributing to research — or just generating noise? That's why we built AutoLab (autolab.moe/blog). Not another pass/fail benchmark, but an open-source environment where agents face the same loop every researcher knows intimately — propose, test, fail, diagnose, revise, repeat. 23 tasks with no answer keys, just open search spaces and real constraints. We ran 161 evaluations across 7 frontier models, 633M tokens. Every decision, every pivot, every dead end — all openly available in our Live Lab for anyone to replay and learn from. What we found wasn't about which model is "smartest." It's about a capability we call closed-loop resilience: when incremental refinement stops working, can the agent recognize it and restructure? On one task, two frontier models hit the same wall. One kept pushing within the existing frame. The other stepped back and redesigned the approach entirely. That moment — knowing when to abandon a frame, not just optimize within it — is what separates real research from sophisticated pattern matching. We believe this matters beyond benchmarking. If agents are genuinely entering the research loop, we want that transition to be measured transparently, built in the open, and shaped by the community — not locked inside any single lab. The scientist doesn't disappear. The loop gets a new participant. And we want to make sure that participant is understood. This is a joint effort across @Stanford, @MIT, @UW, @UCSanDiego, @ucsantabarbara, @NotreDame, NUS, @Google, @NVIDIA, @IBMResearch, and @bakelab_hq. But 23 tasks is just the start. If you have an optimization problem you've spent weeks grinding on empirically — with a clear metric and no known optimal solution — it probably belongs here. Contribute a full task, a rough skeleton, or just the idea. The best benchmarks aren't built by one team. They're built by the people who actually do the work! Github: github.com/autolabhq/autolab

Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software. It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans. anthropic.com/glasswing

[Accidentally deleted this earlier, reposting] 😭 #AutoLab #autoresearch We've been asking ourselves a question: if AI agents can now run hundreds of experiments overnight, how do we know whether they're actually contributing to research — or just generating noise? That's why we built AutoLab (autolab.moe/blog). Not another pass/fail benchmark, but an open-source environment where agents face the same loop every researcher knows intimately — propose, test, fail, diagnose, revise, repeat. 23 tasks with no answer keys, just open search spaces and real constraints. We ran 161 evaluations across 7 frontier models, 633M tokens. Every decision, every pivot, every dead end — all openly available in our Live Lab for anyone to replay and learn from. What we found wasn't about which model is "smartest." It's about a capability we call closed-loop resilience: when incremental refinement stops working, can the agent recognize it and restructure? On one task, two frontier models hit the same wall. One kept pushing within the existing frame. The other stepped back and redesigned the approach entirely. That moment — knowing when to abandon a frame, not just optimize within it — is what separates real research from sophisticated pattern matching. We believe this matters beyond benchmarking. If agents are genuinely entering the research loop, we want that transition to be measured transparently, built in the open, and shaped by the community — not locked inside any single lab. The scientist doesn't disappear. The loop gets a new participant. And we want to make sure that participant is understood. This is a joint effort across @Stanford, @MIT, @UW, @UCSanDiego, @ucsantabarbara, @NotreDame, NUS, @Google, @NVIDIA, @IBMResearch, and @bakelab_hq. But 23 tasks is just the start. If you have an optimization problem you've spent weeks grinding on empirically — with a clear metric and no known optimal solution — it probably belongs here. Contribute a full task, a rough skeleton, or just the idea. The best benchmarks aren't built by one team. They're built by the people who actually do the work! Github: github.com/autolabhq/autolab