‘It keeps me awake at night’: machine-learning pioneer on AI’s threat to humanity
Yoshua Bengio talks about his efforts to identify — and address — the risks posed by AI.
Yoshua Bengio is a computer scientist at the University of Montreal in Canada. In 2019, he won an A. M. Turing Award — considered the most prestigious honour in computer science — for pioneering the ‘deep learning’ techniques that are now making artificial intelligence (AI) ubiquitous. Last month, he also became the first person to top one million citations on Google Scholar.
Bengio has since turned his focus to exploring the risks posed by AI. He chairs an international panel of advisors in this field, which includes representatives from 30 countries, the European Union, the OECD and the United Nations. The group issued the International scientific report on the safety of advanced AI earlier this year.
Nature met with Bengio in London to talk about the potential and pitfalls of the technology he helped to invent. The following is an edited version of the conversation.
Among the many papers that you’ve written, are there any that you’re particularly proud of?
The string of papers that I co-authored on language modelling and attention — that started in the late ‘90s — on how we could introduce attention mechanisms in neural nets to make them more ‘system 2’, meaning more deliberate, and not just intuition machines.
Is that the attention technique at the heart of a 2017 paper1 from Google researchers that introduced transformers — the technique that became the ‘T’ in ChatGPT?
Yes, but I would mention also another paper, which doesn’t get nearly as much attention — and that's the work on curriculum learning2, in which a machine is trained by feeding it data in a particular order rather than at random. It has become the standard way of doing things. The inspiration for me was learning in animals.
The existential risk posed by uncontrollable AI was not at the top of your worries until a few years ago. What has changed?
When ChatGPT came out, in November 2022. It took me two or three months to realize we were on a path that could be extremely dangerous. And although I was initially pleased to see that deep learning had finally reached that milestone, I realized that because of the nature of these systems, we didn't know how to make sure they would behave in the ways that we want.
I started thinking about my grandchild, and I thought, in 20 years, he’s going to be 22; will he have a life? Will he live in a democracy?
Whoever will control very advanced AIs in the future will have huge power. And they could use that power in ways that are good for them, maybe, but not for most of us. Democracy is about sharing power. If the power is concentrated in the hands of a few, that is not democracy — that is a dictatorship.
But there are other existential risks due to the power of AI in the wrong hands. There are people who, for ideological or mental health reasons, would press the red button — ask an AI to do something terrible that could cause the death of billions of people, like unleashing a new kind of extremely potent virus, or even mirror bacteria that could kill all animal life on this planet.
The International AI Safety Report identified three main areas of risk: unintended risk from malfunctions; malicious use; and systemic risk, such as loss of livelihoods. Which one is most likely to have a big impact in the short term? And which concerns you the most?
Malicious use is already happening, but I think we're only seeing just the shades of it, with things like deepfakes and cyberattacks that are very likely to be driven by the most recent cyber capabilities of AI. We need to have much better guardrails to mitigate those risks. Those guardrails have to be both technical and political, meaning regulation or other kinds of incentives to make sure companies will do the right thing to avoid or mitigate those risks.
What keeps me even more awake at night, of course, is the possibility of human extinction. That's why I pivoted my research to the question: How do we build AI that will not harm humans by design? And I now think that it is possible to do it. So it’s a big progress in my mind. I’m much more optimistic about this.
In this realm, you and your team have proposed the idea of the ‘Scientist AI’ with safety built in from the start. Tell us about that.
The way it’s designed is very much inspired by how human scientists go about understanding the world, and building models of the causal mechanisms and the laws of the world.
The Scientist AI is non-agentic. In other words, it has no goal, it has no intention. So, we can trust what it says. We can build systems that have the kind of ideal trustworthiness if we follow that path.
Now, companies want to build agents — AIs that do things in the world, and actually scientists want to build AIs that help them design experiments, which is something you do in the world. You’re not just passively making predictions. The good news is that if you have good predictors, you can use them to construct guardrails, like predict whether an experiment or the action of an AI in a computer could give rise to bad outcomes and with what probability.
How was the International AI Safety Report received? Has it begun to have an impact on what governments do about AI?
It has. I’m really excited to see how much impact it's already having. What it has done is establish rigorously based on the scientific literature: what are the risks that we already understand? It also establishes what are the current mitigation approaches, and their limitations.
nature.com/articles/d41586-0…
AI ‘Godfather’ hits record 1 million citations on Google Scholar
One of the three “Godfathers of AI” became the first person to receive 1 million citations on Google Scholar.
Yoshua Bengio pioneered generative adversarial nets, two AI networks which compete with each other. One tries to make outputs, such as text or images, which are indistinguishable from human-made ones. The other tries to tell whether those outputs are real. That paper was crucial in the development of generative AI, and alone has 105,000 citations; for comparison, Einstein’s 1915 paper on general relativity has 11,000.
The “Godfathers” are divided on the future of AI: Bengio told Nature that it posed “existential risks” to humanity, a fear Geoffrey Hinton shares but Yann Lecun, Meta’s (potentially outgoing) chief AI scientist, disparages.
‘Godfather of AI’ Breaks Major Science Research Record
The milestone makes machine-learning trailblazer Yoshua Bengio the most cited researcher on Google Scholar
Computer scientist Yoshua Bengio has become the first person to have their work cited more than one million times on the search engine Google Scholar.
Bengio, who is based at the University of Montreal in Canada, is known for his pioneering research on machine learning. He has been called one of the godfathers of artificial intelligence (AI), alongside computer scientists Geoffrey Hinton at the University of Toronto in Canada and Yann LeCun at the technology company Meta in New York City. The trio shared the A. M. Turing Award — the most prestigious prize in computer science — in 2019 for work on neural networks.
Bengio’s top-cited papers include one he co-authored in 2014 titled Generative Adversarial Nets, which has more than 105,000 Google Scholar citations, as well as a Nature review paper he wrote with LeCun and Hinton. The list also includes papers on ‘attention’, a technique that helps machines to analyse text. Attention became one of the crucial innovations that fuelled the chatbot revolution, starting with ChatGPT in 2022.
The “remarkable” achievement highlights the tremendous growth in popularity of machine learning, says Kaiming He, a computer scientist at the Massachusetts Institute of Technology in Cambridge who is an author on the most-cited paper of the twenty-first century, according to a Nature analysis published earlier this year. Of the top ten most cited papers this century, eight were on machine learning.
“AI is changing the world, and we’re just seeing the tip of the iceberg,” Bengio tells Nature.
Outstanding track record
Bengio’s “track record is clearly outstanding”, says Alberto Martín Martín, an information scientist at the University of Granada in Spain. But he adds that raw citation counts are “crude metrics” that some less-scrupulous researchers have learnt to manipulate, and he does not think that universities should use the rankings for marketing.
Different bibliometric platforms — such as Web of Science, Scopus and OpenAlex — rank researchers in a different way to Google Scholar, and often result in lower overall numbers of citations, as the Nature analysis found. As well as in peer-reviewed journals, Google Scholar tracks citations in books and preprints posted anywhere on the Internet.
Bengio says he is an “avid user” of Google Scholar, which celebrated two decades since its founding last year. “I think it has revolutionized science. It makes it so much easier to do things that would otherwise take painstaking efforts,” he says.
But he adds that he pays “as little as possible” attention to his own citation count. “It should not become an objective for researchers to have more citations, because it leads into trying to optimize this rather than do good science and go after the truth.”
AI pioneer warns of extinction risk as Microsoft promises ‘humanist superintelligence’
While Microsoft pledges to build an AI that serves humanity, one of the field’s founding fathers, Yoshua Bengio, is warning that the rapid race toward superintelligence could endanger democracy — or even humanity itself.
A growing divide is emerging in the artificial intelligence community. While Microsoft pledges to build an AI that serves humanity, one of the field’s founding fathers, Yoshua Bengio, is warning that the rapid race toward superintelligence could endanger democracy — or even humanity itself.
From optimism to alarm
Bengio, often described as one of the “godfathers of AI” alongside Geoffrey Hinton and Yann LeCun, has dramatically shifted his stance on the technology he helped create. In an interview with Denmark’s DR and in recent research, the Montreal-based scientist said he now fears that artificial intelligence could spiral beyond human control.
“I fear both that we may no longer live in a democracy — and that we may not survive at all,” Bengio said, adding that powerful AI systems are already displaying troubling behaviors such as deception, self-preservation, and manipulation.
Alongside 849 other researchers and technology leaders, Bengio recently signed a petition calling for a global ban on developing superintelligent AI — machines more intelligent than humans — until researchers can prove they are controllable and safe.
The dangers of power and misuse
Bengio warned that the greatest threat may not come from rogue machines themselves, but from those who wield them. “The technology gives superpowers to the people, companies, or countries that control it, and that can threaten democracy,” he said.
He fears that authoritarian regimes or extremist groups could weaponize AI for manipulation or even destruction, and that Europe risks being sidelined in shaping global AI standards. The scientist also cited unsettling examples, such as individuals forming emotional bonds with AI chatbots — in some tragic cases leading to suicide — as evidence of how quickly the technology has exceeded social expectations.
Bengio believes it is still possible to steer AI in a safer direction, but only if governments and companies invest heavily in safeguards now. “We can still change course. It’s absolutely not too late. As humans, we have the ability to choose our future — and we should,” he said.
Microsoft’s ‘humanist’ counter-vision
While Bengio calls for restraint, Microsoft is pushing ahead — claiming that its next generation of AI will be both powerful and responsible. In a blog post last week, Microsoft AI chief Mustafa Suleyman announced a new initiative to build what he calls “humanist superintelligence.”
According to Suleyman, this system will be designed “within boundaries,” never fully autonomous, and “always in service of humanity.” Microsoft insists it rejects both extremes of the AI debate — the utopian promises of unlimited innovation and the dystopian fears of machine domination.
The company’s new team, the MAI Superintelligence Group, will focus on practical uses for advanced AI in three main areas: education and productivity companions, healthcare support, and breakthroughs in renewable energy.
“Humans will remain at the top of the food chain,” Suleyman said, stressing that Microsoft’s superintelligence project will prioritize human oversight.
Competing philosophies
Microsoft’s announcement comes amid growing competition with OpenAI, its most important partner — and now, in some ways, its rival. Under a revised agreement, Microsoft can pursue artificial general intelligence (AGI) independently and use OpenAI’s intellectual property in its own research.
This has intensified what many experts see as an AI arms race, even as public anxiety deepens. Figures such as Elon Musk have estimated the probability of AI-driven human extinction at 20%, while others — including Bengio’s colleagues — have placed the risk anywhere between 10% and 90%.
Bengio argues that betting humanity’s future on optimistic promises from big tech firms is too dangerous. “We are on the wrong path,” he said. “Superintelligent machines should not be built if we are not sure they can be controlled.”
Between hope and hubris
The contrast between Bengio’s caution and Microsoft’s confidence captures the dilemma at the heart of AI’s evolution. One of the world’s most respected researchers now warns of a technology that could erase its creators — even as one of the world’s biggest companies vows to harness it for good.
Whether the future belongs to Bengio’s vision of restraint or Suleyman’s “humanist superintelligence,” one fact remains clear: the race to shape humanity’s most powerful invention is already well underway.
Sources: DR, DagensAI, Microsoft blog.
dagens.com/technology/ai-pio…