Today, we propose the Conditional AI Safety Treaty in @TIME as a solution to AI's existential risks. AI poses a risk of human extinction, but this problem is not unsolvable. The Conditional AI Safety Treaty is a global response to avoid losing control over AI. How does it work?

It is good that the US government decides which models are too dangerous, and restricts access to them. There are many issues with this particular action though: - Models with similar risk levels are not restricted, making this particular regulation arbitrary. It is concerning that the US government seems to act based on haphazardly demonstrated issues, rather than structural risk evaluation. - Not restricting access for US users is unsustainable for a truly dangerous model. Adverse users may also be US citizens, foreign adversaries could easily gain access through US citizens. - Without controlling the hardware supply chain, in particular AI GPUs, regulation may not be enforceable. Governments should start preparing for this. - Some models may get dangerous even before release, e.g. during pre-training, post-training, or testing. Evaluations will need to be carried out at these stages to exclude the relevant risk scenarios.

No comment.

Post-AGI, lack of economic growth is not going to be our main concern. Inequality and crushing our planetary boundaries may be. The ideas of @PikettyWIL, a long-time champion of fighting inequality, should therefore be seriously studied by those trying to make AGI go well.

No comment.

Whether this is corporate propaganda or not: we really do need international regulation (of course not a voluntary agreement!) to verify AI model safety. One option would be our Conditional AI Safety Treaty proposal: x.com/i/status/1857424437680…

The Stop AI trial is finally happening! Wynd Kaufmyn is being prosecuted for blocking the front door to OpenAI's offices. But Wynd argues that her actions were necessary to prevent a greater threat: human extinction. 🧵

It seems a step forward that the Trump administration signed an AI Executive Order! That having been said, almost everything remains to be done: - This EO is voluntary, where we need real, binding AI regulation. - This EO is merely about cyber-risk, while dangerous AI capabilities also include persuasion & manipulation, political strategy, weapons acquisition, long-horizon planning, AI development, situational awareness, and self-proliferation. See our takeoverbench.com for data on how fast these risks are growing! All of them, and possibly more, need to be covered by regulation. - For safety, the least we need is a licensing regime, where unapproved AI may not be released, or not even created, depending on the danger. Such a licensing regime is explicitly not included in this EO. - On current trends, open-weight AI can achieve the same capabilities just a few months later, but without functional guardrails. Regulation will need to address this when capabilities grow to a worrying level, without increasing inequality. - Regulation will need to be internationally aligned, at the very least with China. - Regulation will need to be enforced. Therefore, AI accelerator GPUs, and the supply chain building them, need to be regulated, too. This should start now, since there is a long delay here! In addition, solutions will need to be found for the (many!) non-existential issues AI creates, such as (but very much not limited to) possible mass unemployment, runaway inequality (both inside and between countries), data center greenhouse gas emissions, and many others. If permanently entrenched, these could become existential in their own right. Ever more powerful models will lead to an obvious need for ever more regulation. This is the only option to make AI go well. Politicians delivering what this era needs will, in the end, come out on top.

One of the many things I dislike about the style of "make AI go well" discourse from frontier AI companies is how nationalist the whole thing has gotten. In the 2010s, it was: "we're here to benefit all of humanity" In the 2020s, "we're here to benefit all of 4% of humanity" And even the Good People are buying into this frame completely😢 And they expect humanity to go along, because "come on, be a realistic adult, it's either us or Chiiina" or something like that (And on the EU side, you get "[X] with European values", which too often seems to mean "the same stuff they're doing, but with Us instead of Them in charge") Very big-dog-small-dog energy, morally speaking.

This could be the trial of the century. Sad it isn't being covered as such (yet). "Ms. Kaufmyn argues her actions were necessary to prevent imminent harm from OpenAI’s pursuit of Artificial Superintelligence. Notable UC Berkeley AI professor Stuart Russell is expected to provide expert testimony at the trial. The necessity defense was used in 2018 when a Boston Municipal Judge acquitted 13 climate activists who blocked a fracked gas pipeline. The judge found the activists not guilty on the basis of necessity."

AI should not replace humans, no matter how smart, conscious, or good it might become. Like all animal species, we have a right to live, if desired in our natural, unenhanced form.

Our talk and drink in London this Saturday is almost sold out already! Be quick to RSVP: luma.com/mqc9yxv2

Although Pope Leo XIV's encyclical doesn't explicitly mention the superintelligence that AI companies say they're racing toward, its opposition to it in favor of controllable AI tools is clear: 'Humanity — in all its grandeur and woundedness — must never be replaced or surpassed.' vatican.va/content/leo-xiv/e…

An OpenAI model just produced original research a Fields Medalist says he'd publish in Annals of Mathematics "without any hesitation." If a general-purpose AI is doing the top 0.01% of cognitive work, what jobs are actually safe? Besides, AI is not JUST coming for your job. Pause frontier AI. ⏸

It is obvious that AI will be regulated once it's powerful enough.