In a new AI Snake Oil essay by me and
@sayashk, we do a deep dive into AI existential risk probability estimates. We find that these forecasts are just feelings dressed up as numbers, and even the best-run, well funded, time intensive forecasting efforts result in a range of probability estimates that spans many orders of magnitude! We are forced to conclude that AI x-risk forecasts are far too unreliable to be useful for policy, and in fact highly misleading. We caution against speculation being laundered through pseudo-quantification.
Full essay:
aisnakeoil.com/p/ai-existent⦠(about 7,000 words). Summary below.
Background
Over a year ago we got deep into the AI x-risk literature. We were skeptical but not dismissive. We wanted to identify valid concerns while rebutting bad arguments on their own terms. We've been especially interested in how policymakers should think about x-risk. Today's essay is the first in a series. We've been circulating private drafts for a while and have incorporated a lot of great feedback. I'm excited that we're finally launching this series of essays today!
Analogy: alien invasion
If the two of us predicted an 80% probability of aliens landing on earth in the next ten years, would you take this possibility seriously? Of course not. You would ask to see our evidence. As obvious as this may seem, it seems to have been forgotten in the AI x-risk debate that probabilities carry no authority by themselves. Probabilities are usually derived from some grounded method, so we have a strong cognitive bias to view quantified risk estimates as more valid than qualitative ones. But it is possible for probabilities to be nothing more than guesses.
The reference class problem
The domains where forecasting has been successful, such as geopolitics, rely on the existence of reasonably good reference classes of past events. A reference class for turmoil in one country is turmoil in other country. Reference classes for AI x-risk are things like ... animal extinction. Letās get real. This kind of reference class tells us nothing about the possibility of developing superintelligent AI or losing control over such AI, which are the central sources of uncertainty for AI x-risk forecasting.
Subjective probabilities vary by orders of magnitude
Lacking grounded methods, forecasts are necessarily āsubjective probabilitiesā, that is, guesses based on the forecasterās judgment. Unsurprisingly, these vary by orders of magnitude. Consider the Existential Risk Persuasion Tournament (XPT) conducted by the Forecasting Research Institute in late 2022, which we think is the most elaborate and well-executed x-risk forecasting exercise conducted to date. It involved various groups of forecasters, including AI experts and forecasting experts. The 75th percentile AI expert estimate and the 25th percentile forecasting expert estimate differ by at least a factor of 100.
All of these estimates are from people who have deep expertise on the topic and participated in a months-long tournament where they tried to persuade each other! If this range of forecasts here isnāt extreme enough, keep in mind that this whole exercise was conducted by one group at one point in time. We might get different numbers if the tournament were repeated today, if the questions were framed differently, etc.
It's all speculation
Whatās most telling is to look at the rationales that forecasters provided, which are extensively detailed in the 754-page report. Forecasters arenāt using quantitative models, especially when thinking about the likelihood of bad outcomes conditional on developing powerful AI. For the most part, forecasters are engaging in the same kind of speculation that everyday people do when they discuss superintelligent AI. Maybe AI will take over critical systems through superhuman persuasion of system operators. Maybe AI will seek to lower global temperatures because it helps computers run faster, and accidentally wipe out humanity. Or maybe AI will seek resources in space rather than Earth, so we donāt need to be as worried. Thereās nothing wrong with such speculation. But we should be clear that when it comes to AI x-risk, forecasters arenāt drawing on any special knowledge, evidence, or models that make their hunches more credible than yours or ours or anyone elseās.
Forecast skill can't be measured
We often hear that forecasting has a great track record and so we should trust it. This makes no sense ā why should we trust that someone who is good at forecasting elections or other kind of events is good at forecasting AI x-risk? Besides the fact that these are completely different kinds of events, there just isn't any real evidence to draw upon for the AI x-risk estimation, so being good at finding and weighing evidence is not a skill that helps much here.
Besides, the math doesn't work out. We show that if someone is good at forecasting common events but systematically overestimates rare events, we would have to evaluate them on millions of forecasts before this became apparent.
Summary of the main argument: none of the three probability estimation methods yields reliable AI x-risk forecasts.
Risk estimates may be systematically inflated
There are many reasons why forecasters might systematically overestimate AI x-risk. The belief that AI can change the world is one of the main motivations for becoming an AI researcher. And once someone enters this community, they are in an environment where that message is constantly reinforced. And if one believes that this technology is terrifyingly powerful, it is perfectly rational to think there is a serious chance that its world-altering effects will be negative rather than positive.
And in the AI safety subcommunity, which is a bit insular, the echo chamber can be deafening. Claiming to have a high "p(doom)" seems to have become a way to signal oneās identity and commitment to the cause.
So what should governments do about AI x-risk?
Our view isnāt that they should do nothing. But they should reject the kind of policies that might seem compelling if we view x-risk as urgent and serious, notably: restricting AI development. As weāll argue in a future essay in this series, not only are such policies unnecessary, they are likely to increase x-risk. Instead, governments should adopt policies that are compatible with a range of possible estimates of AI risk, and are on balance helpful even if the risk is negligible. Fortunately, such policies exist. Governments should also change policymaking processes so that they are more responsive to new evidence. More on all that soon.
The full essay is in our newsletter. We plan to publish the rest of the series over the next few weeks. Thank you for reading!
aisnakeoil.com/p/ai-existentā¦