New SOTA on ARC-AGI - V1: 79.6%, $8.42/task - V2: 29.4%, $30.40/task Custom submissions by @jeremyberman and @_eric_pang_ are now the best known solutions to ARC-AGI Both: * Are open source * Use Grok 4 * Implement program-synthesis outer loops with test-time adaptation

btw I've started having doubts about near-term transformative AGI. Timelines up to 2035 or 2040 seem plausible (my mainline scenario is ≤2028 still). We have scant theory of human scientific genius and near zero reliable data on its inner operation. «A genius is just a scaled-up error-corrected human… with extra obsession», perhaps. Something something MCTS, neural noise, branching factor, iterative algorithms; there's no space in the genome for occasionally rolling a superior subspecies with very different mechanics and inductive biases; of course. But is even a properly trained LLM a scaled-anything human, or just an error-corrected interpolation of human reflections? What is the scaling law for the latter towards the genius? And our attempts at making genius obsolete with scale and error-correction procedures have been, I believe, floundering since 1950s-70s; we're coasting on applied science and providing opportunities to geniuses which we do discover in a frenetically expanded talent pool. All papers I've seen about LLMs exceeding humans in "creativity", proving complex theorems or whatnot, discovering this compound or that architecture – have been meh upon scrutiny. Scaling test time compute allows to approximate ever better the flawless work of a mediocre mind that is content to terminate itself in rabbitholes. We are not used to mediocre minds working flawlessly, as in flesh all faults tend to be conjoined. But flawless mediocrity is not genius. Since 2014 I've been a believer in artificial genius, and since 2015 all we've been getting in this regard were amazing demos that can't walk on their own legs towards the summit. Will this change by EoY? By EoY 2025? 2028, surely? On fundamentals, I have to keep predicting "yes, very likely so". But my faith wavers. Galkovsky had this metaphor about a rat being chased into a bloodied corner with a multitude of tools (the rat representing himself, a Soviet citizen somehow incompatible with the regime and society around). A normal rodent is crushed swiftly. A talented one fights back, jumps more adroitly, bites at the fingers that hold the prongs, and perhaps even escapes into some lateral hole to survive another day or week. But a genius rat, if there is such a thing – flies away. Breaks the rules of the rat gridworld, violates the whole theory of the game in which the adversary exists. (Later this was, I think, referenced in Pelevin's Hermit and Six-Toe). No doubt rats cannot fly, and certainly humans can't do super-Turing computations; whatever it is that we can do, our machines can do it too. Yet this is no guarantee that we won't need one more revolution before finally building a machine that is peer to the Gods among us – not merely a clearer reflection of their cast-off shells, still hopelessly beneath it. What this means in practical terms is: we might not get all that many technological breakthroughs within a decade or two even if LLMs and their successors supercharge economic growth. Doordash bots but no cryonics. Automated Walmart but not SpaceX or CERN. It's possible that the world is a tougher nut to crack. You'd probably be wise not to act as if the end is nigh. Do your taxes, raise your kids, study your craft, live – and allow that you will die the way all previous generations have, with much the same means to minimize regret.

MasterMind -- a mini ARC-AGI you can try at home 🔴🟠🟡🟢🔵🟣 Looking for a simple check to compare model reasoning? Play a game of MasterMind with them. Prompt: Let's play mastermind. I pick the code and you guess. We'll play classic rules with 6 colors and duplicates are allowed. Use round colored emojis to indicate your guess. I will answer with black or white round emojis. Before your first guess, display all the emojis we'll be using. The model should always be able to find the answer within 12 turns. Only the very best current models play at "expert" human level. Older open source models like Llama 3.3 70b perform very poorly. DeepSeek R1 goes completely overboard with reasoning. OpenAI's latest 4o, o4-mini and o3 aren't particularly impressive. The best model in my tests is Claude Sonnet 3.7: it solves with few guesses and very compact and fast reasoning. Comparing the reasoning traces between the models is particularly interesting.

When do you think we'll see the first >10% ARC Prize entry on Kaggle?