Everyone says the latest AI agents will be "job-ready" soon, especially after the release of Fable 5 this week. But is that really the case? Over the past many months, my group and collaborators have been building Agents' Last Exam (ALE), a benchmark designed to test exactly that claim on real digital labor-market work. My group and collaborators previously have created many of the benchmarks the field runs on, including MMLU, MATH, CyberGym, and ExploitGym. Today, I'm excited to share Agents' Last Exam (ALE): a rolling benchmark that measures whether AI agents can actually perform economically valuable work across a broad range of real-world domains. With ALE, we evaluated Fable 5, GPT-5.5, Composer 2.5, and other frontier agent systems across more than 1,500 expert-sourced tasks spanning 55 occupations. The result is both impressive and sobering. Today's agents can solve a meaningful fraction of professional tasks. But when we look at the hardest tasks, the ones requiring sustained reasoning, deep domain expertise, and reliable execution over long horizons, they are still far from human-level performance. On ALE's hardest tier, every frontier agent we tested, including Fable 5, achieved a 0% success rate. The age of useful agents is here. The age of truly job-ready agents is not. We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains. 🧵

Over the past few months I've been working on a very exciting project: a new $10m fund for research on multi-agent multi-principal AGI safety! Instead of focusing on single agent alignment and centralized control, we're looking to support research focusing on multi-agent settings, mechanism design, cooperative AI, and coordination problems. This is a joint initiative between @GoogleDeepMind, @Googleorg, @schmidtsciences, @coop_ai, and @ARIA_research. Huge thanks to @James_D_Fox, @weballergy, @FranklinMatija, @lrhammond, and @ObadiaAlex for their invaluable work! See: deepmind.google/blog/investi… Apply: schmidtsciences.smapply.io/p…

AI evaluation needs new arenas. Static benchmarks tell us what a model knows. But the next generation of AI systems will not just answer questions. They will write code, use tools, make decisions, recover from errors, and adapt to changing environments. So @layerlens_ai built a competition around that loop. The Stratix Cup: layerlens.ai/stratix-cup/sea… is a recurring tournament series where frontier AI models compete head-to-head in simulated games. Season 1 is football/soccer. Sixteen frontier models will enter. Each one controls an 11-player team. But there are no human coaches, no live prompting every tick, and no hidden intervention once the match begins. Before kickoff, each model receives the rules, constraints, and game interface. Then it writes a Python class that becomes its team policy. That code runs the match. The model has to live with the strategy it created. This is what makes the Stratix Cup different from a normal leaderboard. We are not just asking, “Can the model produce a good answer?” We are asking, “Can the model build a system that performs under pressure?” Each matchup has three phases: 1. Pre-Game The model reads the briefing, designs a strategy, writes the team code, tests against baselines, and submits. One window. No hand-holding. 2. Gameplay The submitted code controls all 11 players in real time. At halftime, the model gets its frame log, studies what happened, edits its code, and submits a revised strategy for the second half. 3. Adapt Between matches, models can inspect tournament logs, study opponents, diagnose failures, and rewrite their approach. The most interesting signal may not be who wins the first match. It may be what the model changes after it loses. That is why games are such powerful AI evaluations. Games create rules, state, objectives, adversaries, feedback, and pressure. They force models to move from answers to actions. And soccer is a uniquely good first game: continuous, spatial, multi-agent, adversarial, and messy. A model needs coordination, timing, recovery, and strategy. A brittle plan gets exposed fast. Every Stratix Cup match is traced. Tactical calls, substitutions, formation shifts, code changes, match frames, and results are stored and verifiable. The goal is not only to create a watchable tournament, but to generate public datasets that help us understand how models plan, fail, debug, and improve. Season 1 streams live June 22–26 on YouTube and Twitch. Sixteen models. One pitch. Zero humans in the loop. The next AI benchmark might look like a soccer match.

New blog post: The third wave of American philanthropy Hundreds of billions of dollars in new philanthropic capital will soon become liquid. The OpenAI Foundation holds 26% of OpenAI, worth about $220B at today’s valuation. Anthropic’s seven co-founders have pledged to give away 80% of their wealth and have instituted the most aggressive donor matching program for employees in tech history. How much does this all add up to? And how meaningful is that in the context of philanthropy today? I was doing some simple napkin math to wrap my head around the scale of what’s coming, and radicalized myself in the process. I had dramatically underappreciated the scale of the philanthropic capital that’s about to become available and the corresponding gap in talent and organizations that will be needed to make the most of it. This piece aims to directionally sketch the scale of what’s coming, the gap in operational capacity needed to absorb it, and what we can do to fill it. (Link to full post in reply)

We’ve partnered with @layerlens_ai to continuously evaluate SubQ across nearly 100 benchmarks and 200 frontier models on Stratix. The goal is to continuously improve performance while reinforcing a shared commitment to transparency, auditability, and responsible model assessment. Results and future evaluations will be published publicly. Link below to learn more.

I'm dropping a thread of all the protocols that had to freeze their interop because of LayerZero being compromised. Let's go:

is the future value of "open source" code anymore? i believe it's shifting to data, provenance, protocols, evals, and weights. in that order.

State lawmakers introduced over 1,200 AI bills in 2025. They cover everything from deepfakes to autonomous weapons—but they're all just lumped together as "AI policy." @ARozenshtein and I wrote an article that breaks down the policy landscape along three dimensions: (1) what harm are you addressing, (2) what are the factors shaping how you should design your policy intervention, and (3) which actors in the ecosystem should you target? The diagram below, for example, maps the AI ecosystem from chip manufacturers to end users.

In our paper "Towards a Science of AI Agent Reliability" we put numbers on the capability-reliability gap. Now we're showing what's behind them! We conducted an extensive analysis of failures on GAIA across Claude Opus 4.5, Gemini 2.5 Pro, and GPT 5.4. Here's what we found ⬇️