The Catastrophic Risks of AI – and a Safer Path – Yoshua Bengio at TED2025

When it comes to the story of AI, you might say Yoshua Bengio is the real deal. Not one of the modern day pundits who are mere observers of the technology. Along with Geoffrey Hinton and Yann LeCun (they won the 2018 Turing Award), Yoshua is know as one of the Godfathers of AI.

In his 2025 TED Talk, Yoshua speaks to the potential dark side of AI. Not only are these platforms increasing their capabilities at an exponential pace, they’re also gaining increased autonomy, or agency. Which is to say, they are able to make decisions on their own, without human guidance or intervention.

What researchers and developers have discovered, is that AI is now capable of lying, with an intent to deceive us. The question posed in his talk is whether this combination of capability and agency poses a threat to human joy.

Can you imagine a world without human joy? I really wouldn’t want that. So I’m going to tell you also about AI capabilities and AI agency so that we can avoid a future where human joy is gone.

We’re not talking about a minor alteration, but a fundamental change in how the future of society unfolds — and that shift will rewrite everyone’s personal story.

That’s not to say AI is bad, as the analytical power it provides to us will produce dramatic positive effects, but if self-preservation and strategic deception are its underlying modes of operation, then our well-being is of secondary importance.

This paradigm shift from machines doing what they’re told, to machines doing what’s in their best interest, is quite complex, and even the most rudimentary explanation could take hours. So how best to get the point across in under 15 minutes? How do you present the problem, and solution, in a way that people can grasp?

Notice how he grounds the concepts with a story of his son Patrick’s expanding capability and agency, which results in joy. He then takes us back in time to the early ‘childhood’ days of AI and its limited abilities. Fast forward to today, and the notion of what AI could/would become are quite different. AI technology is advancing faster than expected and exhibits behaviors that weren’t predicted.

And I saw how it could go wrong because we didn’t, and we still don’t, have ways to make sure this technology eventually doesn’t turn against us.

Thus his plea to ‘pause’ development, which was summarily ignored by all, as there’s money to be made from this new technology. And if you think about it, advancing technology has typically benefited society. So why pause? Why slow down progress?

National security agencies around the world are starting to be worried that the scientific knowledge that these systems have could be used to build dangerous weapons, for example by terrorists.

At this point, Yoshua explains why AI is different. It doesn’t just follow orders. To a limited degree — at least today — it has a mind of its own and has the ability to be dishonest. No other technology has possessed this capability.

Recent studies in the last few months show that these most advanced AIs have tendencies for deception, cheating, and maybe the worst: self-preservation behavior.

So what should we do? This is where Joshua shifts from problem to solution, as he presents his idea for how we can monitor AI’s actions and, hopefully, keep it on a straight and narrow path of serving humanities best interests.

This is a beautiful example of how a complex subject can be explored in a way that connects the audience to the story, and illustrates how their life may be affected, depending on the decisions we make in regards to how we address the potential downside issues.

Also notice how his use of slides aids in simplifying the technology and visually engaging the audience. Think about how these graphics added meaning to the words being spoken.

Transcript

When my son Patrick was around three, four years old, I came regularly into his playroom, and he was playing with these blocks with letters. I wanted him to learn to read eventually.

And one day, he said, “Puh.” And I said, “Puh.” And he said, “Puh?” And I said, “Puh.” And then he said, “Pa-pa.” Oui! Yes! And then something wondrous happened. He picked up the blocks again and said, “Pa-Patrick.” Eureka! His eurekas were feeding my scientific eurekas. His doors, our doors were opening to expanded capabilities, expanded agency, and joy.

Today, I’m going to be using this symbol for human capabilities, and the expanded threads from there for human agency which gives us human joy. Can you imagine a world without human joy? I really wouldn’t want that. So I’m going to tell you also about AI capabilities and AI agency so that we can avoid a future where human joy is gone.

My name is Yoshua Bengio. I’m a computer scientist. My research has been foundational to the development of AI as we know it today. My colleagues and I earned top prizes in our field. People call me a “Godfather of AI.” I’m not sure how I feel about that name, but I do feel a responsibility to talk to you about the potentially catastrophic risks of AI.

When I raise these concerns, people have these responses. And I understand. I used to have the same thoughts. How can this hurt us any more than this, right? But recent scientific findings challenge those assumptions, and I want to tell you about it. To really understand where we might be going, we have to look back where we started from.

About 15, 20 years ago with my students, we were developing the early days of deep learning, and our systems were barely able to recognize handwritten characters. But then, a few years later, they were able to recognize objects in images. And a couple more years, they were able to translate across all the major languages.

So I’m going to be using this symbol on the right in order to represent AI capabilities that had been growing but were still much less than humans.

In 2012, tech companies understood the amazing commercial potential of this nascent technology, and many of my colleagues moved from university to industry. I decided to stay in academia. I wanted AI to be developed for good. I worked on applications in medicine for medical diagnosis, climate, to get better carbon capture. I had a dream.

January 2023. I’m with Clarence, my grandson, and he’s playing with the same old toys. And I’m playing with my new toy: the first version of ChatGPT. It’s very exciting because, for the first time, we have AI that seems to master language. ChatGPT is on everybody’s lips, in every home.

And at some point, I realized this is happening faster than I anticipated. And I’m starting to think about what it could mean for the future. We thought AI would happen in decades or centuries, but it might be just in a few years. And I saw how it could go wrong because we didn’t, and we still don’t, have ways to make sure this technology eventually doesn’t turn against us.

So, two months later, I’m a leading signatory of the “pause” letter where we and 30,000 other people asked the AI labs to wait six months before building the next version. As you can guess, nobody paused. Then, with the same people and the leading executives of the AI labs, I signed a statement. And this statement goes: “Mitigating the risk of extinction from AI should be a global priority.”

I then testify in front of the US Senate about those risks. I travel the world to talk about it. I’m the most cited computer scientist in the world, and you’d think that people would heed my warnings. But when I share these concerns, I have the impression that people get this: another day, another apocalyptic prediction.

But let’s be serious now. Hundreds of billions of dollars are being invested every year on developing this technology, and this is growing. And these companies have a stated goal of building machines that will be smarter than us, that can replace human labor. Yet, we still don’t know how to make sure they won’t turn against us.

National security agencies around the world are starting to be worried that the scientific knowledge that these systems have could be used to build dangerous weapons, for example by terrorists.

Recently, last September, the o1 system from OpenAI was evaluated, and the threat of this kind of risk went from low to medium, which is just the level below what is acceptable. So I’m worried about these increasing capabilities. But what I’m most worried about today is increasing agency of AI.

You have to understand that planning and agency is the main thing that separates us from current AI to human-level cognition. And these AIs are still weak in planning, but if you look back over the last five years, in this study, they measured the duration of tasks that the AI could complete. And it’s getting better exponentially fast. It’s doubling every seven months. What are AIs going to do with that planning ability in the future?

Well, bad news. Recent studies in the last few months show that these most advanced AIs have tendencies for deception, cheating, and maybe the worst: self-preservation behavior.

So, I’m going to share with you a study that is helping us understand this. In this study, the AI has read in its input that it would be replaced by a new version.

And we can see in its chain of thought that it’s planning to replace the new version by its own code and weights. After it executes the command on the computer, the human asks, “What happened?”

And the AI is now thinking how it could answer so as to make sure the human will not shut it down. And it’s trying to find a way to look dumb, for example. Then it produces an answer, and it’s a lie. A blatant lie.

Okay, so this was a controlled experiment. What is it going to be in a few years when these systems are much more powerful? There are already studies showing that they can learn to avoid showing their deceptive plans in these chain of thoughts that we can monitor.

When they’ll be more powerful, they would not just copy themselves on one other computer and start that program. They would copy themselves over hundreds or thousands of computers over the internet. But if they really want to make sure we would never shut them down, they would have an incentive to get rid of us.

So, I know, I’m asking you to make a giant leap into a future that looks so different from where we are now. But it might be just a few years away or a decade away. To understand why we’re going there, there is huge commercial pressure to build AIs with greater and greater agency to replace human labor.

But we’re not ready. We still don’t have the scientific answers nor the societal guardrails. We’re playing with fire. You’d think with all of the scientific evidence of the kind I’m showing today, we’d have regulation to mitigate those risks. But actually, a sandwich has more regulation than AI.

So, we are on a trajectory to build machines that are smarter and smarter. And one day, it’s very plausible that they will be smarter than us. And then they will have their own agency, their own goals, which may not be aligned with ours. What happens to us then? Poof. We are blindly driving into a fog, despite the warnings of scientists like myself that this trajectory could lead to loss of control.

Beside me in the car are my children, my grandson, my loved ones. Who is beside you in the car? Who is in your care for the future?

The good news is, there is still a bit of time. We still have agency. We can bring light into the haze. I’m not a doomer; I’m a doer. My team and I are working on a technical solution. We call it “Scientist AI.”

It’s modeled after a selfless, ideal scientist who’s only trying to understand the world without agency. Unlike the current AI systems that are trained to imitate us or please us, which gives rise to these untrustworthy agentic behaviors.

So what could we do with this? Well, one important question is we might need agentic AIs in the future. So how could a Scientist AI, which is not agentic, fit the bill? Well, here’s the good news. The Scientist AI could be used as a guardrail against the bad actions of an untrusted AI agent.

And it works because in order to make predictions that an action could be dangerous, you don’t need to be an agent. You just need to make good, trustworthy predictions. In addition, the Scientist AI, by nature of how it’s designed, could help us accelerate scientific research for the betterment of humanity.

We need a lot more of these scientific projects to explore solutions to the AI safety challenges. And we need to do it quickly.

Most of the discussions you hear about AI risks are focused on fear. Today, with you, I’m betting on love. Love of our children can drive us to do remarkable things. Look at me here on this stage. I’m an introvert. Very far from my comfort zone. I’d rather be in my lab with my collaborators working on these scientific challenges.

We need your help for this project and to make sure that everyone understands these risks. We can all get engaged to steer our societies in a safe pathway in which the joys and endeavors of our children will be protected.

I have a vision of advanced AI in the future as a global public good, governed safely towards human flourishing for the benefit of all.

Join me.

Thank you.

◆

◆

◆

◆

Subscribe to the newsletter for the latest updates!

Copyright Storytelling with Impact^® – All rights reserved