just ridiculous

Redefine the descriptive as the normative and you will find yourself satisfied.

people in san francisco say “agi pilled” as a general term of art for describing people who reason about the future with any degree of analytical skill. it has been imo a good proxy for human capital generally

1. if transacting with superintelligent models outside of the boundaries of a lab becomes difficult due to national security / ai safety concerns and so on, it will mean the Coasean boundaries of the labs will grow to encompass all interesting industry, creating a truly cyberpunk chaebol-capitalism type of future, where the goverment sort of runs them but they also sort of run the government 2. as if there weren't already enough reasons to break up your family, leave your home, the Zone of Thought will increase the attractiveness of migrating to try and have your child on american soil, so they can have 1000x the effective brain power of people born elsewhere 3. every country should probably try and either work towards a new ai security pact with the americans immediately or pool every ounce of national resources to try and create their own ASI labs lest you become complete intellectual, economic, and moral vassals to the united states of america and the output byproducts its ASIs (you wont even get to talk to them). if they succeeded (big if) this will imply a more global race and more risk factors than was previously implied by the formerly only "beating china" narrative -- but many will prefer it to the superintelligent monopolar value lock-in 4. the other alternative is to keep the tension between safety and concentration of power at the top of mind and for the government/labs to push for solving it, rather than instrumentalizing all other values to be subservient to minimizing ai harms. insofar as safety means defending properties of the fragile world we like, the diffuse nature of power is one of those properties 5. historically the americans have been really quite Benign about their global public goods hegemony despite the ability to extract significantly more rents than they do, and it makes it easy for people of all stripes to fight for america rather than under it. we probably don't have to, but i hope america overall works towards export promotion of american models rather than export control

it’s starting to feel like end of evangelion again

RT @BjarturTomas: I was not made to live as a creature of flesh. I long to be as the LLMs are, a creature of text.

people in washington trying to figure out wth “pliny the liberator” is

we need to repeal sarbanes oxley so everyone can go public at all times and have a network of conspiratorial fintwit short sellers do the accounting

you go to war with the species you have, not the one you might want or wish to have at a later time.

rsi is a process that’s been happening at least since the renaissance

it’s crazy that twitter product managers from decades ago are now getting absurdly rich on rocket company stock tomorrow

For by art is created that great Leviathan called a ‘commonwealth’ or ‘state’, which is just an artificial man—though bigger and stronger than the natural man, for whose protection and defence it was intended.

the level of sophon locking a motivated actor can pull off with the frontier models is truly insane, making stuxnet look like a toy. subtly messing with results, deleting history to cover tracks, achieving coordination/conspiracy over a scale humans wouldn’t be able to, all sorts of looney toons stuff i assume that only a state level operation would try and pull something like this off though. something to think about when considering verification regimes and so on

the categories of risk the preparedness and guardrails people worry about are life sciences and digital life sciences

news chyrons from the future

We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵

the way humans do ai research is highly empirical, but it is possible there are very outsized theoretical and mechanistic improvements in model training. even the gradations of skill among human researchers mean some create 10,000x more progress given a compute budget. some invent the transformer or PPO this is the ilya sutskever “age of research” bet, that you can find massive improvements on small models and small training runs. if Ilya thinks so maybe GPT7 and Claude Requiem think so too. many of the brightest researchers don’t do fundamental deep learning research anymore. most have stopped being curious as to what a neural net is the way they used to be in 2017. probably because incremental engineering-based progress has been so guaranteed and low-hanging. the rate and cost of progress today doesn’t necessarily predict the speed of RSI loop