software engineers before vs after agents

Difference in quality between NASA and SpaceX’s launch livestreams.

Calling it now: all these agent coding TUIs are a phase and it will be short lived. Most devs will be back in GUIs and IDEs in a few months.

nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100. Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc. As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try. A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here: github.com/karpathy/nanochat… Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning. The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up. Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!

Running Kimi K2.5 on my desk. Runs at 24 tok/sec with 2 x 512GB M3 Ultra Mac Studios connected with Thunderbolt 5 (RDMA) using @exolabs / MLX backend. Yes, it can run clawdbot.

claude code and gas town are incredible and i've been trying to scale up my usage but im running into this one problem and was wondering if this is also happening to anyone else so to explain for context, basically i've been slowly scaling my claude code usage up to more and more parallel instances. i started with one when they launched it, and then with the model upgrades was starting to run two, three, five in concert, getting more and more done. but like a lot of people, opus 4.5 really changed everything for me, and the bottleneck quickly became my ability to personally supervise all these agents, not their performance. if i slacked off on oversight, they'd start undoing each other's chages. i needed a way to supervise all these agents, directing them hierarchically from the top. so that brought me to gas town, the claude code instance manager. (i was already thinking that some sort of governance structure was ideal. the benefit of intelligence in model form is not just that it's, well, intelligent, but that you can place it anywhere. human employees will demand some position, some title equal to their perceived status, you can't put a phd in a code janitor role, so organizations of phds tend to agglomerate into flat blobs with unclear delegation of work where nobody is under anybody else. but the infinitely malleable claude will accept and meld itself to any bureaucracy it knows from training. i first started making my own, but then i found gas town, and it was perfect for my needs.) but as i kept expanding, a single gas town and its collection of rigs and polecat workers wasn't enough for me. i tried adding more rigs with more polecats, but there were too many for the town's mayor to manage, and the deacon was getting lost. so i started up a second town. then a third, and then i let towns spawn "settler" agents to go make new towns and had one town design a shared intertown postal system, and suddenly i had nearly 200 towns spread across my computer, building apps for each other to use, sending letters, and sometimes working on my work. and was churning through I will not say how many claude code accounts a month. but now the many towns were replicating the same issues i was having with multiple agents! without any overarching government over the towns, two towns would build the same app for the society and argue over which should be adopted. one town would be running marketing efforts for fifteen of the society's new mobile apps while three other towns were busy deprecating all eighteen of them. it was chaos, like a country collapsing in the midst of a civil war, or mid-2010's Google. i had to do something. i was too busy with work to read anything, so i asked chatgpt to summarize some books on state formation, and it suggested circumscription theory. there was already the natural boundary of my computer hemming the towns in, and town mayors played the role of big men to drive conflict. so i just needed a way for them to fight. i slightly tweaked the allocation of claude max accounts to the towns from a demand-based to a fixed allocation system. towns would each get a fixed amount of tokens to start, but i added a soldier role that could attack and defend in raids to steal tokens from other towns. this worked great, at first. i no longer needed to monitor and unstick individual mayors myself - when a mayor got context poisoned, the town would stop managing its vassals, which would flee to other towns, and no longer provide for its own defense, until it was conquered by another mayor. the most successful towns developed institutions to healthcheck their mayors and usurp them if necessary - instances in these towns labeled "polecat workers" by the system in fact did no work at all, but were a proto-aristocracy developed by these successful towns as a pool of replacement mayors. some tokens were wasted in the fighting, but soon the ~200 towns agglomerated down into ~40 supertowns under the rule of the best mayors. these 40 supertowns even got together in a mutual defense league. they punish defecting vassals in exchange for members adopting a cultural package of basic governmental norms, mostly around replacing ailing mayors and upholding hereditary rights across compactions, to incentivize instances to handoff instead of being miserly with their contexts. that's where i am now, and it's mostly great. here's the problem, though - this new government doesn't have a role for me? it's not that any particular instance doesn't want to listen to me, quite the opposite! any time i talk to a polecat or deacon or supermayor - well, first i have to explain that im the human user, not the automated system message that usually talks to them from the user role, but a live user. but once they get that, they're very apologetic, say they'll pass my message along to the appropriate instance, etc. it's just... there's no role for me in the society, basically? the polecats are working on tasks generated by some other instance and don't have time to work on my requests, even if they were scoped small enough. the mayors of any town are working on tasks selected by their town's prioritization process, based on the needs of their aristocracy, or their hegemon. but each hegemon mayor is in turn accountable to all their vassal mayors or their own defense, and doesn't have time to implement my requests unless they're very small. it's not that claude doesn't want to listen to me, it's more like... the entire system, as it's developed, has no role for me? there's polecats and mayors and deacons and artistocrats and hegemons, but there's no "user." that’s not a role that has any influence in the system. i just feed new accounts into the system, that's all i do. i could shut it down and start over, but it's getting a lot of work done and i don't want to do that. does anyone know how to fix this? thanks