Not the line of questioning I was expecting in a hearing about Chinese IP theft, but I'm glad senators are starting to really get why "we must beat China!" is really nowhere near a complete plan for how to make sure AI goes well. --- Transcript: HAWLEY: Ms. Toner, can I just come back to something that I think you said in response to Senator Durbin. You said to him that, regarding American AI companies, you said that it is hard to believe but nevertheless true that American AI companies are working as hard and as fast as they can to try to develop technology that will displace many millions of workers and potentially pose existential risks. Now that's my gloss, maybe you wanna correct the record exactly as you said it before. I thought that was very interesting and very important. Could you just reiterate that for us? TONER: Yes. AI is a very fast-moving field, and I think it is important that as we think about what AI's implications are for our society, for our civilization, we don't merely look at the AI systems that we have today—chatbots, starting to be agents that can help a little bit with some professional tasks—but instead we take seriously the goals of the companies that are building these systems. Over the past 10 or 20 years, it's gone from a very abstract idea that we might build AI that can outperform humans at any intellectual task, to a pretty concrete idea that some of the most well-capitalized companies in the history of the planet are driving towards as fast as they can. They may fail! It may turn out to be harder than they think to build systems that are that capable. Personally, I'm skeptical of some of the extremely short timelines that they name, saying we might have these superintelligent AI systems within, you know, one to three years. But it seems so clear that there's a real possibility that they build these systems within three years, 10 years. If they build it within 10 years, that's when my daughter is entering high school. That's not very long. That is an extremely radical thing to be trying to do, to build computer systems that can outperform humans, that may escape the control of humans, and the companies are telling us they're doing it, and I think we don't take them seriously, and we should. HAWLEY: These same companies often say, and often in front of this committee and to this body, that it's absolutely vital that they succeed at whatever it is they're doing on that particular day, in order so that we can beat China. You know, they're our great American national champions and we have to beat China. My concern is based on what you've just testified to and what I've heard others testify to, it sounds an awful lot like the goals that they have in mind, that these companies, these CEOs have in mind, are every bit as nefarious. In fact, if these same goals were held by a foreign adversary, we would say this is an incredible threat to our national security, we'd never allow a foreign corporation to try and pursue such plans at the expense of American workers, at the expense of American families, and yet these companies, our own companies so to speak, are doing it. Let me just ask it this way: Will it do us any good if these American AI companies are able to pursue their designs without any hindrance? Will it do any good that we beat China if in fact they succeed in displacing millions of American workers, gobbling up all of Americans' data, completely destroying our IP system, etc.? TONER: I think the way I've heard this put best is: Right now, the way that we build AI and the level of control we have over it, which is not great, the winner of any AI race between the US and China is the AI. And I think we need to be working to make sure that is not the case. I think it is very important that the US AI sector remains ahead of the Chinese AI sector, but if that's at the expense of AI overrunning the entire planet, then that is, you know, that hasn't benefited us. HAWLEY: Yeah, that sounds entirely sensible to me and I just have to say I don't really have any interest in winning an AI race in which the goal, the victory rather, the prize for success is to become like China. Is to become a surveillance state. Is to become a place where there is no private property any longer, where nothing is personal, nothing can be protected, nothing can be owned by any individual. Why in the world would we want that in the United States of America? I mean, if the prize is to destroy everything that makes us Americans, why would we compete in that game? It seems very dangerous to me. Let me ask you something else about competition with China though. You also testified to Senator Durbin that the best way, if I remember correctly, the best way to constrain China's ability to match us in AI development is to constrain the hardware to which they have access. That seems to be an important point to me, can you just elaborate? TONER: Yes. I think there's different levers of what goes into having a competitive AI ecosystem, and many of them, talent, data, algorithmic ideas, are very difficult to control. We're very fortunate that we're in a situation where the most advanced hardware is produced by American companies, is designed by American companies. And I think we, if you look at the, China is growing their capacities here, but they're not growing them nearly fast enough to meet their own domestic demand, nor are the US companies to be clear. So we can control chips to China and not forgo any profits, not forgo any revenue because the demand for those chips is so great. I'll also call your attention to semiconductor manufacturing equipment, what goes in the fabrication facilities. I think it's even more strategically clear that we should not be allowing China access to advanced tools. That is something that has gotten lip service from the past three administrations but enforcement has been very weak. And I think ensuring that the most advanced lithography tools, the most advanced design software, other aspects of the semiconductor supply chain are not being exported to China to let them build their own indigenous supply chain is also one of the simplest and most important levers we have available. HAWLEY: Let me just conclude by saying that I think it is absolutely vital that we bend this technology, this AI technology which is upon us whether we like it or not, that we bend it to the good of the American worker and the American family. And I am firmly of the view that this is not just going to happen magically. That if we just stand back and just wait to see what will happen, it's not going to be good for American workers, it's not going to be good for American families. We've got to make a choice as a society to make it so. And this is the time to make that choice right now. youtube.com/watch?v=KHmo9vGQ…

I hate to say it but an international agreement between the US and China to ban superintelligence is inevitable. Leaders in these countries are just going to follow their incentives, and none of them are willing to give up control to an artificial superintelligence.

Every AI lab is working to make their AI helpful, harmless and honest. Max Harms (@raelifin) thinks this is a complete wrong turn, and 'aligning' AI to human values is actively dangerous. In his view a safe AGI must have absolutely no opinion about how the world ought to be, be willingly modifiable, and be entirely indifferent to being shut down. The opposite of all commercial models today. The key appeal is that so-called 'corrigibility' could be an attractor state – get close enough and the AI actively helps you make it more corrigible over time. That forgiveness would at least give us a shot. It's a strategy that feels natural within the 'MIRI worldview', recently laid out by his colleagues @ESYudkowsky and @So8res in 'If Anyone Builds It Everyone Dies'. But it risks causing a different AI catastrophe, because the resulting AI model would necessarily be willing to assist any human operator with a power grab, or indeed any crime at all. I interviewed Max on the 80,000 Hours Podcast to debate the MIRI worldview, and what we should do to figure out if corrigibility ought to be our one and only focus. Links below – enjoy! 00:01:56 If anyone builds it, will everyone die? The MIRI perspective on AGI risk 00:24:28 Evolution failed to ‘align’ us, just as we'll fail to align AI 00:42:56 We're training AIs to want to stay alive and value power for its own sake 00:52:24 Objections: Is the 'squiggle/paperclip problem' really real? 01:05:02 Can we get empirical evidence re: 'alignment by default'? 01:10:17 Why do few AI researchers share Max's perspective? 01:18:34 We're training AI to pursue goals relentlessly — and superintelligence will too 01:24:51 The case for a radical slowdown 01:27:53 Max's best hope: corrigibility as stepping stone to alignment 01:32:34 Corrigibility is both uniquely valuable, and practical, to train 01:45:06 What training could ever make models corrigible enough? 01:51:38 Corrigibility is also terribly risky due to misuse risk 01:58:57 A single researcher could make a corrigibility benchmark. Nobody has. 02:12:20 Red Heart & why Max writes hard science fiction 02:34:08 Should you homeschool? Depends how weird your kids are.

If you think that's weird, check out all the strange bedfellows who would prefer not to have a global thermonuclear war!

A tale of two warning shots, #1: COVID happened. Scientists are divided on whether it was a lab leak. The world did not rally against dangerous viral research in labs. The warning shot was squandered.

"The most urgent film of our time." THE AI DOC: OR HOW I BECAME AN APOCALOPTIMIST is only in theaters March 27. Watch the trailer now.

Reminder: Donations to MIRI before Jan 1 are high-leverage. We’ve got ~$1.6M in 1:1 matching from SFF, over half of which has yet to be claimed! This is real counterfactual matching: whatever doesn’t get matched by the end of Dec 31, we don’t get. 🧵 intelligence.org/2025/12/01/…

the lightcone needs you to lock in Apply to the 2026 MIRI Technical Governance Team Research Fellowship

I recommend this work by Oscar Delaney, which revisits the idea that a superpower pursuing ASI could be deterred by (the threat of) sabotage.

We at the MIRI Technical Governance Team just put out a report describing an example international agreement to prevent the creation of superintelligence. 🧵

Short video on the call for a superintelligence ban. (Now past to 68,000 signatures!) youtube.com/watch?v=pSlzEPnR…

If AI improves fast, that makes things worse, but it's not where the central ASI problem comes from. If your city plans to enslave ultra-smart dragons to plow their fields and roast their coffee, some problems get *worse* if the dragons grow up very quickly. But the core problem is not: "Oh no! What if the huge fire-breathing monsters that could wipe out our city with one terrible breath, that are also each individually much smarter than our whole city put together, that when mature will think at speeds that make any human seem to them like a slow-moving statue, *grow up quickly*? Wouldn't that speed of maturation present a problem?" If you imagine suddenly finding yourself in a city full of mature dragons, that nonequilibrium situation will then go pear-shaped very quickly. It will go pear-shaped even if you thought you had some clever scheme for controlling those dragons, like giving them a legal system which said that the humans have property rights, such that surely no dragon coalition would dare to suggest an alternate legal system for fear of their own rights being invalidated. (Actual non-straw proposal I hear often.) Even if you plan to cleverly play off the dragons against each other, so that no dragon would dare to breathe fire for fear of other dragons -- when the dragons are fully mature and vastly smarter than you, they will all look at each other and nod and then roast you. Really the dragon-raising project goes pear-shaped *earlier*. But that part is trajectory-dependent, and so harder to predict in detail in advance. That it goes grim at *some* point is visible from visualizing the final destination if the dragons *didn't* revolt earlier, and realizing it is not a good situation to be in. To be sure, if dragons grow up very fast, that *is* even worse. It takes an unsolvably hard problem onto an even more unsolvably hard problem. But the speed at which dragons mature, is not the central problem with planning to raise n' enslave dragons to plow your fields and roast your coffee. It's that, whether you raise up one dragon or many, you don't have a dragon; the dragons have you.

One of the best interviews so far. The book launches on Tuesday! semafor.com/article/09/12/20…

AI companies are currently actively trying to build smarter-than-human AI. If they succeed, then every man, woman, and child on Earth is probably going to die. This is actually happening. I, Robby Bensinger, am genuinely scared for myself, my loved ones, and the rest of you over the next two years. If we have more than fifteen years, I'll consider us lucky. Nobody knows how far off this technology is, but there's an insanely huge and well-funded effort to build it, and senior researchers in AI generally agree that we're probably only 2 or 5 or 10 years away, not 20 years away. The non-profit I work for is rushing out a book to try to urgently alert policymakers and the public about the situation, so that the international community has a chance of responding quickly enough. We've gotten some incredibly strong endorsements, including: - Jack Shanahan, a retired three-star general and the inaugural director of the Pentagon’s Joint AI Center, the coordinating hub for bringing AI to every branch of the US military. - Bruce Schneier, one of the most prominent computer security experts in the world. - Jon Wolfsthal, the Obama administration's senior nuclear security advisor. - Suzanne Spaulding, the former head of the DHS Cybersecurity and Infrastructure Security Agency, the US government's main agency for cybersecurity and critical infrastructure security. Questions welcome. If you want to help: go to ifanyonebuildsit dot com to preorder the book. Reshare this image and get your friends / families / communities to preorder this. Have conversations about this and get the word out there. The book is our main vehicle for trying to inform as many journalists, public figures, and policy people as we can, as fast as possible. Preorders have an outsized impact on how many people hear about the book, via impacting bestseller lists and print run sizes. This is actually happening, and we need your help.

Some huge book endorsements today — from retired three-star general Jack Shanahan, former DHS Under Secretary Suzanne Spaulding, security expert Bruce Schneier, Nobel laureate Ben Bernanke, former US NSC Senior Director Jon Wolfsthal, geneticist George Church, and more!

Announcing: IF ANYONE BUILDS IT, EVERYONE DIES by co-founder of @MIRIBerkeley @ESYudkowsky and MIRI president @So8res is coming September 16th. This book is an urgent warning about the looming risk of extinction from artificial superintelligence. ifanyonebuildsit.com

New AI governance research agenda from MIRI’s Technical Governance Team. We lay out our view of the strategic landscape and actionable research questions that, if answered, would provide important insight on how to reduce catastrophic and extinction risks from AI. 🧵1/10

I've now participated in four daylong workshops over several years on the intersection of AI and Nuclear Command, Control, and Communications (NC3). Most of the discussion is about whether and how to incorporate AI into NC3. But given what I've heard about how long changes to NC3 take, and the current pace and direction of progress, I anticipate that we will develop strong AGI (by whatever definition one likes) well before any substantial changes are made to the US NC3 system. That leaves several uncomfortable questions that I would direct toward US national security folks: Uncomfortable question 1: Do you believe that the current US NC3 system is robust against attack by an adversary that is much smarter than any collection of humans, and operates at 100x or more human speed? Uncomfortable question 2: If not, are you prepared to cede human control over the defense of the US NC3 system to a similarly superhuman US AI system, which may itself be uncontrollable? If not, what is the plan? Uncomfortable question 3: Given the above two questions, do you believe that the development of such AI systems by the US's adversaries presents an existential threat to US national security? Uncomfortable question 4: By the same logic, do you believe that the development of such AI systems by the US presents an existential threat to Chinese and Russian national security, at least in the eyes of China and Russia? How do you anticipate these countries would react to believing this existential risk is imminent? Uncomfortable question 5: Given that many experts believe that this strength of AI may be inherently uncontrollable, do you believe that the development of such AI systems even by US companies may present an existential threat to US national security? Uncomfortable question 6: Given the above, do you believe it is in the national security interest of the United States to allow the development of AGI, domestically or abroad?