All existing evidence points to the fact that our best-capitalized approach to AI development (scaling RL, SFT) requires the persistence of what
@dwarkesh_sp very aptly summarized as a "massive black hole of data".
The exponential bros will never stop trying to convince you that RSI is a discontinuity that's right around the corner and that once the system sees/does enough to "improve itself", this massive black hole of data (and the "temporarily" anti-competitive, Orwellian, centralized behavior it requires to persist) will be replaced by non-rivalrous access to aligned intelligence that's too cheap to meter.
Don't worry guys, we are only temporarily restricting your ability to do AI research with our models! We're only temporarily altering performance based on who you are. Once the model is smart enough to iron out the wrinkles, AGI is a public good, a TFP subsidy, with equal access for all!
Unfortunately, the only way we currently know to create something approximating a super intelligent system (one that can do everything better than anyone or anything) is by showing it everything! Real-world processes have irreducible time constants: you can't simulate the real world faster than real time.
Therefore, to remain on the "frontier" in the way we think about it, the labs will either have to 1) incentivize or 2) coerce the continuous economy-wide production of the work data that makes these real world processes legible to their clusters.
At present, the labs are using capital to "incentivize" this production and access: coding agent token subsidies (give us your engineer's decision traces), "deployment" joint ventures (let us turn your PortCo into an RL environment and we'll guarantee you 18% returns, preferential access to models, etc.), $2M in tokens for YC startups (we want traces from the firms with the fastest search/learning rates), Thrive Holdings etc.
But its important to remember that the capital itself is born of coercion (not durable long term economic profits).
The story is twofold: 1) if you don't give me the capital, China will win and turn the world into their gulag, and 2) is the Pascal's wager: if there is any possibility that this system can be built it's strictly irrational for you to not be a believer and owner. As capabilities increase, the coercive power of this narrative only accelerates. But these arguments become a lot less charismatic if capabilities stall.
Absent a deliberate alternative architecture this creates increasing pressure toward centralized control over the domains where models are trained through deployment. And this is why the political economy of AGI is beginning to reject its development.
To prevent the political economy from rejecting its development you either have to find a way to build a superintelligence without one entity seeing (and doing) everything in the economy or you have to change the political economy to be amenable to centralization. It's becoming clearer and clear than the labs, not having been able to come up with a good mechanism with which to capture durable value from a decentralized superintelligence, are forced to move to coercion and centralization.
The sad thing is, this is all for naught. We will never beat China if our most well capitalized approach (âgeniuses in a datacenterâ) is implicitly contingent on centralization. Contingent because our current conceptualization, to succeed, fundamentally requires coercing the production of ever larger amounts of legiblized economic dark matterââthe data black holeâ and sending it to the centralized entity with the compute necessary to train âmachine godâ.
Chinaâs political economy allows for centralization that ours never will (without a Civil War). The $295B cluster is just the start. And we have no clue what kind of crazy coercive data generation they are engaging in.
So if we continue down this path, we will lose, unless you believe what Darioâs proposingâyou will live in the pod, eat the bugs, use the only AI, and live on UBIâis charismatic enough to change Americaâs political economy. In this world, the economy will by definition have to become a gulag. An RL environment for machine god. Where your unique understanding of local reality becomes an input to a system that has the implicit goal of replacing you.
I also happen to believe that this approach will create a strictly inferior "superintelligence", we know nature favors bottom's up emergence to top down coercion.
Meta is focusing on creating a generically "SOTA" system, when really they should be focused on creating a SOTA system that arises naturally from their unique strengths. This is why employees feel like they are in the gulag: Meta's trying to turn work into a Gosplan for Cognition. Soon if we are not careful, the labs will try to do the same for the entire economy.