In our scenario, frontier-model export controls were a 2028 event. It took two days for it to happen 😢. The most capable US models are now legally off-limits for non Americans. The silver lining is that it's getting very hard for European policy makers to keep ignoring this.

In medieval times, within the arms race of ever more demonic torture devices, some sadistic genius came up with the idea of the Little Ease. This was a prison cell built so small in every dimension that a grown man could not stand upright in it nor lie down at full length nor properly sit. The pain is relentless and without relief and inflicted by one's own body. Prisoners were known to go insane within a few days. A stay at the Little Ease was considered even more cruel than the rack, the thumbscrew, and the other ghoulish machinery of the Tower of London. A breeding pig will spend her whole life in a version of that box. These are social, roaming creatures (more intelligent than dogs) who will never leave this corset of steel. They have been selectively bred to be bigger than their frames can support. Yet we put them in cells so confined that they cannot comfortably sit, and their attempts to do so (for example, by sneaking their limbs into adjacent stalls) reliably lead to fractures and sprains. They cannot sweat, yet have nothing to roll around in to cool themselves off. Except their own manure, which (contrary to the common misconception) they are so averse to (thanks to their strong sense of smell) that new sows will often suffer from constipation to avoid soiling the space from which they eat and sleep. Here is how the writer Matthew Scully described what saw at one of Smithfield’s “gestation barn”: > “Sores, tumors, ulcers, pus pockets, lesions, cysts, bruises, torn ears, swollen legs everywhere. Roaring, groaning, tail biting, fighting, and other “Vices,” as they’re called in the industry. Frenzied chewing on bars and chains, stereotypical “vacuum” chewing on nothing at all, stereotypical rooting and nest building with imaginary straw. And “social defeat,” lots of it, in every third or fourth stall some completely broken being you know is alive only because she blinks and stares up at you … creatures beyond the power of pity to help or indifference to make more miserable, dead to the world except as heaps of flesh into which the [insemination] rod may be stuck once more and more flesh reproduced.” — The Save Our Bacon Act is trying to unroll the few state protections we have against this barbaric cruelty - for example California’s Prop 12 - which banned the sale of pork from pigs kept in gestation crates. It’s incredibly important we don’t end up with this sort of federal preemption. SOB will not only kill the most important animal welfare related laws in the US of the past decade, but more importantly, it will also restrict ALL future legislative progress (aka how the animal welfare movement has gotten its biggest wins). The Senate is currently deciding whether to add the SOB Act to the Farm Bill. With relatively little money now, we can discourage the most pivotal senators in the Ag committee from backing this amendment. Defeating this bill is even more important given the amount of philanthropic funding I expect to come online in the next year or two. It will plausibly be over 10x more expensive to repeal SOB than to prevent it from passing in the first place. All that money that could be spent transforming our society's relationship to mass animal suffering will instead have to be spent just getting us back to where we are right now. That's why money spent now fighting this bill (and I mean right NOW) is so effective. If you’re in a position to donate six figures, please DM me.

Our highest and most urgent national priority should be AI safeguards. The risks of AI weapons, pathogens, mass unemployment, surveillance, and even extinction must not continue to be largely ignored.

Sad to see Ted Chiang resorting to such bad arguments in this piece. He confidently claims Claude has no inner experience. But he has to use a lot of dodgy philosophy and poor reasoning to get there: 1. We can't take deflationary mechanistic descriptions of how AI calculations are performed to show that AI isn't conscious. Otherwise we could argue that 'humans are just neurones transmitting signals one after another' and thereby conclude humans can't be conscious. But that would be wrong for us. And the same logic could be wrong for LLMs. 2. That LLMs are asked to play characters, and effectively are always playing characters, doesn't mean they aren't conscious. It's true a human playing the role of Caesar doesn't have Caesar's experience of things. But they still experience something (that of being a person pretending to be Caesar). The same could be true of Claude. (Arguably it's also true that humans are always playing characters to some extent and don't have a completely fixed nature, but that has no bearing on our own subjective experience.) 3. Chiang says "an LLM is a machine that generates only one word at a time". This conflates two things: they output one word at a time, and they only think about one word at a time (without planning ahead or looking back). The first is true of AI but equally true of humans. While the latter we know is a false description of how AIs think – we can see from how AIs compose poetry that they plan out rhymes a at least one line ahead. 4. He argues that because it's implausible that basic autocomplete on your phone is conscious, it's similarly implausible that Claude is conscious. Using the same logic we could say that if we feel confident a fruit-fly isn't conscious we can be confident a human being can't be either. A human brain and fruit-fly brain share some information transmission and processing mechanisms in common. But humans do it much more, and do it differently. And those differences may be what makes the difference. Similarly the many types of internal information processing that occur in Claude's weights but not in autocorrect may be exactly the things that get you subjective experience. 5. Chiang confidently claims you need a body to have subjective experience without much argument. He may turn out to be right but the claim is speculative and contested. 6. Chiang leans on the idea that moral reasoning is necessarily subjective/emotional with very little argument, while ignoring competing theories like rationalism. He may be right but moral sentimentalism is a highly contested position that can't simply be assumed. 7. He argues that it would be impossible to convince him that a video of an astronaut around Alpha Centauri was real, because of the surrounding contextual understanding. And similarly no AI output could convince him that Claude is conscious. But we can dismiss the first video as almost certainly fake because we mechanistically understand space travel and physics well enough to know a human couldn't have gotten there in time for it to be real (unless our model of the world were very wrong, which we think is much less probable than a fake video which would be entirely unsurprising). But by contrast we don't mechanistically understand how subjective experience arises. So we simply can't make the same highly confident move of interpretation there. (It's actually the archetypal thing in the universe we perhaps understand least well!) That said, AI outputs barely move my estimate of AI consciousness because they could indeed have been generated by an unconscious process (or not, we just don't know). 8. He argues that "Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript." This is misguided because A. Microsoft Word as a program replicates much less of what humans are functionally capable of than Claude so the argument by functional analogy is basically not present there. B. Files of text don't have any computations going on in or as part of them, even when 'open' in a text editor. They are static. So they have even less in common with what appears distinctive about the human brain, which is constant calculation. So the case by mechanistic or functional similarity is weaker still. Not to mention that neural nets have more in common with the architecture of the human brain than ordinary computer programs, and are grown organically in a way normal software is not. Common sense says says Claude has more in common with a human brain than Microsoft Word or a text file. Common sense is right. So the prima facie case for Claude being conscious is naturally stronger (even if you think it's still weak in absolute terms). ——— I agree with Chiang that looking at the text outputs of LLMs alone won't be enough to make us confident they are conscious. We will need to look at how they work, figure out more about how humans and other animals work, and ideally solve the hard problem of consciousness (!). But none of that licenses us to dismiss out of hand the possibility that LLMs do have subjective experience.

💡 Another round of Longview Philanthropy’s digital minds request for proposals is open for applications. A year ago I would have called this niche. Now AI labs publish model welfare research, public discussion of digital sentience is growing, and the field is expanding. 📈

My best interview in some time. Rohin Shah leads AGI alignment/safety at DeepMind. And he has a lot of spicy personal takes: We probably won’t get catastrophic misalignment (00:49) Safety 'commitments' have severe limitations (10:38) The intelligence explosion probably isn't imminent (1:52:44) Why he's not working to pause AI advances (51:44) Pre-deployment evals aren't the right focus (for catastrophic risks) (37:41) Signalling concern for safety sometimes diverts resources from actually making AI safe (01:09:51) Reading AI thoughts is v useful for safety – and we'll probably be able to for years to come (54:17) Governance is somewhat more likely to be the bottleneck than alignment (43:55) Rohin's team doesn't have a veto, and that's OK (27:36) Central banks are a promising model for regulating AI (33:34) Also: Google DeepMind's actual plan for building AGI safely (1:40:29) How external researchers can positively influence big AI companies (2:21:55) The roles GDM most needs to hire for (2:37:03) On the 80,000 Hours Podcast. Links below - enjoy! (@rohinmshah)

Yeah i'm sympathetic to all of this. But some pushback. I expect that your subjective values will prefer to hand off to AIs that embody better reasoning and open-mindedness than humans, and that are less selfish. That would in practice be a way for you to get more of what you subjectively want and deny others of what they want. That sounds wrong. But if most ppl wanted to create optimised suffering, you'd probably endorse stopping them. Or, less extreme, if they wanted to just leave the universe as a nature preserve. You're not a cultural/ethical relativist: your subjective values don't put "respect everyone else's opinions" above all other ethical considerations. So "push for handing off to ethical AI" might be pretty similar to "push to prevent leaving the universe as a nature preserve". Both go against what other humans want on reflection, but give you more of what you want. It's legitimate for you to push for that outcome. Of course, you shouldn't stage a coup! And you shouldn't lie, manipulate, etc to get your way. But you're allowed to push for what you want in transparent, legitimate ways. Articulate a vision; convince others of it. That's part of public discourse. A natural mechanism would be, once alignment is fully solved, to build ethical agentic AIs and give them economic/political rights over things outside of the solar system (matching human rights). Then market forces will result in those AIs controlling most resources longterm. I do see the cooperative appeal of just pushing for CEV. But it doesn't seem like a dick move to push for something that is better by your personal ethical lights.

Disappointing to see this. We need curious and humble exploration of the complex Qs around ai understanding and consciousness, not overconfident sweeping proclamations. And some of these claims are blatantly false. AIs clearly do understand the work they produce.

In 2016, we said AI and pandemics were the world's most pressing problems. Which aged better than I hoped. Today I still rank AI alignment top, but it's joined by some even more neglected issues. The article tells the whole story.

Full automation of AI R&D likely yields a large speed up even if it doesn't get faster and faster (aka a software-only singularity). This is from: - One-time speed-ups - Additional compute now directly improves researcher quality via smarter AIs I say more in a new post:

New post: Will we really put data centers in space? Read it here: forethought.org/research/wil…

Your career is your most important decision. Especially for your impact. But most career advice sucks, so millions waste it. I’ve spent 15 years researching how to find the best career. This book sums up everything I’ve learned. It feels like my life’s work. And it’s out now.

80,000 Hours the book was published by Penguin today. It's our definitive explanation of how to use your career to try to force the world onto a better track. The result of 14 years honing our ideas. See what's in there and order: 80000hours.org/book/

Still the world's most underrated chart: People think the best charities save 50% more lives than average. The data and experts say it's 10,000%.

So so excited for 80,000 Hours the book to be out, by my cofounder @ben_j_todd - a culmination of 15 years' work! What career you pursue is among the most important decisions in your life, and this is the best guide there is. 80000hours.org/book/

I’m building a multifamily office for this population, where philanthropic capital will be managed as seriously and creatively as investment capital. We’re not at “public launch” stage yet, but since Nan’s tweet is doing numbers I wanted to put up a signal flare.

The world is about to be massively constrained on finding altruistic talent. If you're hungry to use your career to reduce suffering and keep people safe, there's never been a better time to get ambitious, connected, and caught up. Here's what I'd do:

New blog post: The third wave of American philanthropy Hundreds of billions of dollars in new philanthropic capital will soon become liquid. The OpenAI Foundation holds 26% of OpenAI, worth about $220B at today’s valuation. Anthropic’s seven co-founders have pledged to give away 80% of their wealth and have instituted the most aggressive donor matching program for employees in tech history. How much does this all add up to? And how meaningful is that in the context of philanthropy today? I was doing some simple napkin math to wrap my head around the scale of what’s coming, and radicalized myself in the process. I had dramatically underappreciated the scale of the philanthropic capital that’s about to become available and the corresponding gap in talent and organizations that will be needed to make the most of it. This piece aims to directionally sketch the scale of what’s coming, the gap in operational capacity needed to absorb it, and what we can do to fill it. (Link to full post in reply)

Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.

I've spent the last 15 years of my career researching how to find the best career. This book is the culmination of everything I've learned. It's called 80,000 Hours, and it launches in a week.