Flunked out of med school, trying the AI thing. On Bluesky as neurogarg.bsky.social

Joined November 2019
52 Photos and videos
Pinned Tweet
It was recently shown that reasoning models can't avoid forbidden words in their CoT, controllability as low as 0.1%. But is that just a surface-level slip, or does the thought itself persist? I tracked semantic trajectories through embedding space to find out. đŸ§”
1
1
5
521
Pranjal Garg retweeted
Repeatedly Asking Chatbots “Does She Like Me Back?” Gives Them Generalized Anxiety Disorders PDF: immaterialscience.org/s/Chat

2
11
783
Pranjal Garg retweeted
This is the type of slop blowing up the number of submissions to conferences, screwing up the review process and wasting reviewers' time, and furthering the stochasticity of acceptances.
Such a great evening to start a brand new research for NeurIPS in 3.5 days.đŸ§˜â€â™‚ïž Day 1: planning. Night 1: running experiments and sending the abstract. Day 2: reading results fighting with Claude, and sending again. Night 2: sleep (optional). Day 3: opening Codex, and finally, write the pape in parallel. Night 3: resolving the “beef” with Claude (temporary peace) and going to sleep. Day 4: final reading, last-minute fixes, submission then some relaxation, maybe a beach walk. I’ll keep you posted on the results. This will be my only single-author paper, so I can’t hide behind other submissions if it gets rejected 😅
12
14
421
37,792
Pranjal Garg retweeted
The irony is that the person who wrote this post got AI to write it so didn't put the cognitive effort into it ... the collapse is clearly happening!
MIT's Nobel Prize-winning economist just published a model with one of the most alarming conclusions in the AI literature so far. If AI becomes accurate enough, it can destroy human civilization's ability to generate new knowledge entirely. Not gradually degrade it. Collapse it. The paper is called AI, Human Cognition and Knowledge Collapse. Authors: Daron Acemoglu, Dingwen Kong, and Asuman Ozdaglar. MIT. Published February 20, 2026. Acemoglu won the Nobel Prize in Economics in 2024. He is not a doomer blogger. He is the most cited economist of his generation, and his models tend to be taken seriously by the people who set policy. Here is the argument in plain terms. Human knowledge is not just a collection of facts stored in individuals. It is a living system that requires continuous reproduction. People learn things. They apply them. They teach others. They build on prior work to generate new work. The entire engine of science, medicine, technology, and innovation runs on this cycle of active human cognition. What happens when AI provides personalized, accurate answers to every question people would otherwise have to learn themselves? Individually, each person is better off. They get correct answers faster. They make fewer errors. Their immediate outcomes improve. But they stop doing the cognitive work that sustains the collective knowledge base. Acemoglu's model shows this produces a non-monotone welfare curve. Modest AI accuracy: net positive. AI helps at the margin, humans still do enough learning to sustain collective knowledge, everyone gains. High AI accuracy: net catastrophic. AI is accurate enough that learning yourself feels unnecessary. Human learning effort collapses. The knowledge base that AI was trained on is no longer being refreshed or extended. Innovation stalls. Then stops. The model proves the existence of two stable steady states. A high-knowledge steady state where human learning and AI assistance coexist productively. A knowledge-collapse steady state where collective human knowledge has effectively vanished, individuals still receive good personalized AI recommendations, but the shared intellectual infrastructure that enables new discoveries is gone. And the transition between them is not gradual. It is a threshold effect. Below a certain level of AI accuracy, society stays in the high-knowledge equilibrium. Above that threshold, the system tips. And once it tips, the collapse is self-reinforcing. Because the people who would have learned the things that would have pushed the frontier forward never learned them. And the AI cannot push the frontier on its own. It can only recombine what humans already knew when it was trained. The dark irony at the center of the model: The AI does not fail. It keeps giving accurate, personalized, useful answers right through the collapse. From the individual's perspective, nothing looks wrong. You ask a question, you get a correct answer. But the collective capacity to ask questions nobody has asked before, to build the frameworks that generate new knowledge rather than retrieve existing knowledge, that capacity is quietly disappearing. Acemoglu has been the most prominent mainstream economist skeptical of transformative AI productivity claims. His prior work found that AI's actual measured productivity gains were much smaller than the technology industry projected. This paper is a different kind of warning. Not that AI will fail to deliver promised gains. But that if it succeeds too completely, it will undermine the human cognitive infrastructure that makes long-run progress possible at all. The welfare effect is non-monotone. That is the sentence worth sitting with. Helpful until it is not. Beneficial until it crosses a threshold. And past that threshold, the same accuracy that made it so useful is precisely what makes it devastating. Every student who uses AI instead of working through a problem is a data point. Every researcher who uses AI instead of developing intuition is a data point. Every generation that grows up with accurate AI answers and no incentive to develop deep domain knowledge is a data point. Individually rational. Collectively catastrophic. Acemoglu proved this is not just a cultural concern or a vague anxiety about screen time. It is a mathematically coherent equilibrium that a sufficiently accurate AI system will push society toward. And there is no visible warning sign before the threshold is crossed.
6
1
12
2,766
It was recently shown that reasoning models can't avoid forbidden words in their CoT, controllability as low as 0.1%. But is that just a surface-level slip, or does the thought itself persist? I tracked semantic trajectories through embedding space to find out. đŸ§”
1
1
5
521
I primarily built on the work of @jcyhc_ai @tomekkorbak @rmcc_11 @BruceWLee2 and more people. I am always looking for any feedback.
1
1
150
Pranjal Garg retweeted
Interesting experiment: recruit subjects with no biology experience, give half of them access to frontier AI models (circa mid-2025), and ask all of them to perform a series of tasks necessary to create a virus. LLM group had a leg up in most tasks, but were not more successful in the job of actually creating the virus. This is the o-ring theory of automation in action. Many jobs require all tasks to be completed successfully in order to generate viable output. If even one of those tasks is not automated, it may be enough of a bottleneck to prevent significant increase in productivity. In this case, LLMs improved performance on 4/5 tasks, but no improvement on one (molecular cloning). This resulted in no significant differences between the Internet only vs. Internet LLM group. open.substack.com/pub/active

8
27
174
34,678
Pranjal Garg retweeted
A lot of the current discourse about AI comes from a fatalistic position of total surrender of agency: "tech is moving in this direction and there's nothing anyone can do about it" (suspiciously convenient for those who stand to benefit most) But in a free society, we get to choose what kind of world we live in, independent of technological capabilities. Just because tetraethyllead made engines run more efficiently and saved money didn't mean we were *obligated* to pump it into the lungs of our kids Technological determinism is BS. We have a collective duty to make sure AI adoption improves the human condition, rather than hollows it out
121
280
1,789
72,805
Pranjal Garg retweeted
Introducing Willow, our new state-of-the-art quantum computing chip with a breakthrough that can reduce errors exponentially as we scale up using more qubits, cracking a 30-year challenge in the field. In benchmark tests, Willow solved a standard computation in <5 mins that would take a leading supercomputer over 10^25 years, far beyond the age of the universe(!).
2,837
12,000
75,263
19,246,732
Pranjal Garg retweeted
We think that all memory is stored in the brain. But our study published today in @NatureComms shows that all cells—even kidney cells—can count, detect patterns, store memories, and do so similarly to brain cells. My first (co)corresponding author paper!đŸ§”nature.com/articles/s41467-0

315
4,219
24,586
8,463,712
Pranjal Garg retweeted
8 Oct 2024
I feel like the critique that the physics Nobel is going to computer science misunderstands the culture of physics. They aren’t ceding the prize to cs, they’re claiming it as a branch of physics
41
113
1,129
134,944
Pranjal Garg retweeted
Physicists think AI is physics. Statisticians think AI is statistics. Mathematicians think AI is mathematics. Psychologists think AI is psychology. Neuroscientists think AI is neuroscience. And they’re all right.
289
840
5,369
352,743
Pranjal Garg retweeted
According to Nobel, computer science is a branch of physics. So logically, then, mathematics is a branch of literature
177
483
4,835
195,079