#7 Combined Print & E-Book Nonfiction #8 Hardcover Nonfiction

1. Our agreement permits inference, broadly, to continue. It only limits AI training above 10^24 FLOP. In Article VIII, we establish that the agreement restricts only research that advances toward ASI or undermines verification. We permit medical diagnostics and drug discovery, for example, among other non-general capabilities. As long as the research focuses on a narrow domain (like materials science or climate modeling) and does not increase "general cognitive capabilities," it is not subject to restrictions. There is no seizing of assets, but rather mandatory monitoring which can involve forced consolidation of compute into monitored datacenters by the signatory government domestically. (No international centralization of AI monitoring!) So, for example, a small university lab’s research compute would be relocated to a monitored datacenter but is still owned and (remotely) operated by that lab. 2. We architected our agreement to specifically not ban the additional production and installation of compute. The value of such for inference and real-world economic activity is only increasing. So the total economic impact on parts of the AI supply chain are probably far less than existential. We note that many of the relevant players have multi-year backlogs to clear at full production capacity, and building out the infrastructure for model development is only a part of that. 3. Nothing happens to consumer AI apps. If anything additional resources are freed up which can accelerate adoption and integration of these tools. 4. Individual governments handle their own verification and monitoring, but some transparency, random inspections, etc. to either China or the US will be required. Our agreement opts for comprehensive domestic monitoring and shares the least information with rivals (directly) which establishes required confidence. There is no multilateral international AI inspection apparatus in our current version of the agreement, which is a change from the first version we published. 5. Robotics and other applications are unaffected to the extent that progress can continue without developing new models with runs that exceed the compute limit. 6. Our agreement does not attempt to set up a “binding” constraint, in that we do not consider the legal, reputational constraints on US/China to be the active ingredient which leads to stability of the treaty. Instead, we establish a clear understanding on both sides of what behavior is restricted and what consequences must follow otherwise, with enough transparency and non-interference with intelligence gathering to create confidence in each side that the other is adhering to the agreement. Rather than legal constraints, I consider deterrence as the framework for thinking about the stability of the agreement. This is the historical method for constraining a major power, including the use of strategically significant assets. Nuclear deterrence prevented Soviet employment of its conventional superiority against Western Europe. A Soviet dead hand blocked what was otherwise the possibility of a perfect first strike (nuclear or otherwise). Stability emerges when both sides coordinate to make the consequences of aggression clear. Is that a “binding constraint” that one side “accepts”? No, not legally, textually. Not in the sense of “I staked my reputation and my honor in the arena of international relations”. Where agreements have a role to play is in helping to maintain the conditions for successful deterrence. An example of this is the anti-ballistic missile treaty, in which both sides limited their deployment of missile defenses so that deterrence could be maintained.

Godfather of AI (and world's #1 most cited scientist) announces his support for a coordinated global AI pause!

That there are no urgent hearings on Capitol Hill, no serious legislation in the pipeline, and no persistent questioning of candidates for higher office on their proposed approaches to AI is incredible given how transformative the technology is and how fast it is moving.

now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development

'Some people believe that if machines decide to kill us it's the right thing to do because they're smart' AI researcher Nate Soares reveals that some factions in Silicon Valley mistakenly believe that if an AI is exceptionally intelligent it must also be highly moral. @So8res @Freddygray31

This just passed 2M views! If you haven't seen it yet, check out this AI In Context video on "If Anyone Builds It, Everyone Dies" youtu.be/Nl7-bRFSZBs?si=-re5…

Ten places where Magnifica Humanitas matters for AI. At 42k words long, Pope Leo XIV’s new encyclical has a lot to say. In our most recent Digest, Mitchell Howe outlines the parts which might be the most impactful.

What will be the impact of AI industry super PACs? "The takeaway here is that this year’s U.S. midterm elections are being aggressively shaped by different factions of the AI industry sometimes supporting the same candidates, sometimes different candidates, buying ads that don’t have anything to do with AI."

If you're interested in creating videos about the extinction threat posed by superhuman AI, consider applying to this bootcamp!

Our report focuses on claims that are (1) solidly defensible and (2) generally agreed within METR. Here I’ll give some personal opinions on how we should feel about the state of AI risk, and the IMO most important limitations of the report.

"Gathering information is perhaps an important step forward, but it's not nearly enough." In today's Digest, Joe Rogero discusses the new executive order from CA governor Gavin Newsom.

In today's Digest: * OpenAI, Anthropic, SpaceX race to file IPO. * The AI executive order is postponed. * METR evaluates rogue deployment risks. * AI makes a breakthrough in mathematics.

New paper from MIRI data scientist @robi_rahman: Does Distributed Training Undermine Compute Governance?

The wait is over! Starting today, May 15, you can stream @theaidocfilm on @peacock. This film takes the dizzying complexity of AI — the promise, the peril, the competing ideologies, the economic incentives — and creates a shared experience we can all see and respond to. Then, after you've watched, head to humanetech.com/ai-roadmap to explore concrete actions we can all take to build a better future with AI.