space exploration technologies might be the hardest company name of all time. insane cultural victory for tech that the company with the most definite optimist vision of the future is the biggest ipo in history

There’s a big, under-appreciated reason why people may have very different experiences and opinions about using AI for work — are they using it for tasks they’re already an expert at, or tasks they can’t do themselves? The former leads to a *growth cycle* and the latter leads to a *dependence spiral*. When I use AI to do something I’m an expert at, like coding, I treat it as a tool. I can build quickly, maintaining an understanding of the code, knowing that if necessary, I can fix the code myself. It feels empowering. It frees up my time to think about the complex, judgment-oriented parts of software engineering that I can’t or won’t delegate to AI. That means my own skills improve rapidly, and I get to climb the ladder of complexity and develop higher-level skills, much more so than when I write the code myself. I feel in control. I can lock in and achieve a flow state — when AI is working, I’m reviewing, building understanding, and planning the next steps. I never get the feeling that the tool is about to replace me. This is the growth cycle. (Of course, the growth cycle is not automatic. I still need to exercise agency to use AI responsibly. But it’s the same challenge with any productivity-enhancing technology, and those who’ve navigated such transitions before are well-equipped to navigate it with AI as well.) On the other hand, if I use it for tasks I don’t understand and haven’t learned to perform myself, I have no choice but to treat it as a superintelligence. If something breaks, the best I can do is ask AI to fix it and hope for the best. I generally can’t evaluate the quality of the output myself. The only way to find out if it's any good is if and when the work is ultimately reviewed by an actual expert. The experience is confusing, unsettling and disempowering. And forget about flow state. By over-relying on AI, I risk losing whatever skill I had at the task in the first place, even if it boosts productivity in the short term. This is the dependence spiral. It’s no wonder that entry-level workers and students preparing to enter the workforce find themselves in a bind. To compete with the AI-enabled productivity of more seasoned workers, they must adopt AI themselves, but doing so risks the dependence spiral. I have some thoughts on solutions that I will share in later posts, but I think having a clear diagnosis of the problem is a useful first step.

The issue isn't the existence of safeguards. It is that: - the classifier is terrible, exceedingly trigger happy, unusably so for many - silently degrades responses if it's about AI - captures all user data These are *actively* bad, not just a mistake. There are real tradeoffs in safety, and this release chose none of those. It basically nerfed the model in the most blatant way possible, taking none of the nuances into account. AI safety researchers can't use it. Bio researchers can't use it. Cybersec researchers can't use it. Even by the system cards own admission this isn't in "immediately develop superweapons" territory, which makes it even more egregious. They did it because they can. Which invites scrutiny, how can you trust anthropic to do the right thing when it counts? We just had this argument about their fight with DoW. That anthropic didn't want to be the final arbiter, just wanted safety. This is the opposite, they really do want to be the final arbiter.

Sorry. I got to say this publicly. I really agree with @karpathy point here. My wife @mioana is a leading economist and we discuss this all the time. The singularity think of the AI community is rather misguided.

Scientific research is fundamental to advancing civilization and helping people globally to solve the most critical problems, from medicine to materials, from brain science to physics, and much beyond. This is only possible when scientists have access to the best tools of the time to conduct scientific research, including having access to AI-based tools.

Imagine building a computer and not allowing its use in CS research. Thats some dystopian shit.

Degrading performance on ML research *without telling the user* is shockingly hostile and a terrible look. That could silently damage all sorts of work, including some of my own. Also the type of thing that could raise the eyebrows of antitrust enforcers worldwide.

you're not dreaming big enough

Many such cases

I am confused as to why it is surprising that throwing oodles of test time compute can increase the solvability horizon (modulo a good verifier in the loop).. wasn't this known even from as far back as depth-limited minimax?

There are two AGI problems, the 0 to 1 problem (building an intelligent machine > some capability threshold) and the 1 to n problem (building heterogeneous artificial intelligences). Solving the first doesn’t necessarily solve the second.

And a series of autocatalytic, but not recursively self-improving, improvements can have massive economic consequences. The 1st, 2nd, and 3rd industrial revolutions, printing, urbanization, agriculture.

Appreciate Anthropic’s intellectual honesty in their observed autocatalytic effects. Autocatalytic does not presage RSI. Computers, compilers, and the Internet all had autocatalytic properties.

A pause continues to be utter and complete nonsense and it always will be. 1. Let's make planes safer by not making planes! 2. What exactly happens in a "pause"? Do labs get to keep working and we all just get to sit on our hands waiting for this eureka moment? I guess someone gets to keep their checks while we just bring the economy to a crashing halt based on a few people's vague ideas about some imaginary future problems that they came up with while huffing glue and reading Dune! 3. Who will fund the labs when they are not putting out new products? I guess VCs will continue to just give them 100s of billions out of the goodness of their hearts! 4. What actually justifies a pause? Apparently, the wonderful world of imagination! Theoretical future problems that haven't happened yet. Like massive job losses! Um, jobs are increasing including in the areas most affected by AI, like coding, so I guess not that. So what? I know the jobs apocalypse is coming because I can imagine it and imagination is reality, right? Maybe advanced AI weapons? Ah, so the government will pause weapons research too? Well no, they will keep doing that anyway because they always do that. It's what governments do. Okay so we're going to ban Chat Bots while the government keeps making weapons? That should solve everything we're worried about with AI! Well then what about recursively improving models that grow to superintelligence overnight? Yeah that's not really a thing. That's the plot of an Avengers movie. Models are bound by the same real world constraints we are like compute (brains/chips) and time (will this drug have side effects in twenty years can only be known in twenty years) and fuzzy multiplicity (not right or wrong but right-ish and wrong-ish means you can't make a reward signal for "is this the right decision for my business") and the subject/object paradox (the thing improving is judging its own improvement. Yeah chew on that one for a bit.) But I imagined AI overcoming every real world constraint instantly so it's true! 5. How would we know we did everything we needed to do in a pause? How do we know when it's over? We dont. We just want a pause now because we want it! Don't you see my pause ⏸️ emoji? It's nice right! 6. Who gets to decide we are ready to unpause? The government or the people! Great, because vague ass platitudes like this always go well for concrete policy design. Looks, none of this is real. It's theater. It's not real policy. It has no basis in reality. It's a mass hallucination. It's pushed by people who believe in magic and magical solutions. And anything that comes out of it will do infinitely more damage than the imaginary thing they were trying to protect us from in the first place.

I guess I would say 2 things. First, it’s not like we know we have a dominant design yet. Second, there are foundational parts of the human cognitive repertoire that haven’t been cracked via scaling and there is no reason to be optimistic they will. Innovation remains imperative,

You’d be shocked by how many people in think tanks/academia/government/“strategic classes,” including in the U.S., are convinced that Chinese models are now “good enough” and leading the world in adoption. Meanwhile, the reality I see is a fairly wide, and still widening, gap.