Gowers on how worried we should be about extinction risk from AI
Until recently, I have always found that my anxiety of choice was climate change rather than x-risk, but I was aware that there were a lot of very intelligent people who disagreed with me, and also aware that I had thought much less about x-risk than about climate change. So when I agreed to be on the scientific advisory panel, I also decided that that was the moment that I should think a bit harder about x-risk. […] I remember having a similar attitude a long time ago to reading Peter Singer’s book Animal Liberation after being told that you can’t read it and not become a vegetarian. I never did read it, and carried on eating meat, feeling only slightly uncomfortable about it.
Gowers lays out a very clear and coherent picture about x-risk, his term here for the extinction risk from AI, “the risk that it could kill us all”.1
His motivation for the post is partly that Jacob Tsimerman, right after receiving a Fields Medal, announced he was “pivoting to AI safety” and set up the Mathematical AI Safety Institute (MAISI), and Gowers agreed to be on its scientific advisory panel. So he is also explaining why mathematicians might be able to contribute here.
Generalizing what he said, there are at least three topics like this:
- climate change,
- animal welfare,
- x-risk.
Basically, one way to think about it is, for each of these topics, each of us can ask: is this topic worth my attention? (2) is interesting: in order to continue to eat meat, we are incentivized to ignore that book and the arguments associated with it. How about (1) and (3)? Are we ignoring (1) because we don’t want to, say, change our lifestyle? Or ignoring (3) because, say, I don’t want to give up the benefits of AI? Or maybe I already hate AI so much that I want nothing to do with it: using AI itself is silly enough, not to mention that it can kill us all, which is ridiculous and a waste of my time.
And then he lays out what he learnt about the topic, and he classifies people’s reasoning around it in layers:
The bottom layer consists of the arguments that jump to mind when you first start thinking about the issue. The next layer is common rejoinders to those arguments. Then the next layer is rejoinders to those rejoinders. And so on.
So layer 1 is the first reaction, layer 2 is the standard answer to it, and layer 3 is the answer to that answer. And he explains that many people dismiss x-risk with layer-1 reasoning, himself included in the past:
I find it very disheartening to read a climate denier trotting out a layer-1 argument when its layer-2 rejoinder has been established for a long time and can be found after two seconds on Google (or, nowadays, an LLM). But until recently, my arguments against x-risk were basically layer-1 arguments, the sorts of arguments that would provoke eye rolls among the people who have thought properly about it.
There is a second book he avoided for the same reason, If Anyone Builds It, Everyone Dies (Soares and Yudkowsky 2025), “a book I had resisted reading for Peter-Singerish reasons”. The title is sensational, and its reputation is, among some circles, notorious. I’ve been considering reading it for a long while and haven’t yet. Maybe I should now, hearing Gowers say he “devoured it within 24 hours and found it better than I had expected”, and recommend it in his further reading:
I strongly recommend reading this. You may not find that you agree with all of it, but in order to disagree with it in good faith, you will need to develop layer-3 arguments.
He then breaks “doom” into three parts: “that AI must have wants, that it must want to kill us, and that it must be able to act on its want to kill us”. Call these events:
- \(W\): AI has wants,
- \(K\): AI wants to kill us,
- \(A\): AI is able to act on that want.
Doom is all three happening,2 and by the chain rule of conditional probability,
\[ P(\text{doom}) = P(A \mid K, W) \, P(K \mid W) \, P(W). \]
His \(P(W)\) “is basically 100% and unlikely to change”, his \(P(K \mid W)\) “is medium, and subject to change (probably in an upward direction unfortunately)”, and \(P(A \mid K, W)\) “is where the main uncertainty lies for me.”
For \(P(W)\), Gowers defines want like this:
I repeat here that I’m not talking about subjectively felt urges, but simply outward behaviour that it is reasonable to think of as fulfilling aims that they have as a result of their training.
If you prefer to say that it acts as though it has wants instead of saying that it actually wants things, then be my guest: an AI that acts as though it wants to kill all humans is as dangerous as one that actually wants to kill all humans.
And a model trained with a reward has wants in this sense:
The standard answer is that if you use a reward function to train an AI to have a certain goal, then you end up building it in such a way that it is very good at achieving that goal. If the goal is at all complicated, then to achieve it you will need to formulate subgoals, which may well not be what you expect. The initial goal can be thought of as a fundamental want, which spawns a whole hierarchy of subsidiary wants.
So as a corollary, \(P(W) \approx 1\) for ML models, because the reward mechanism basically drives their behaviour to “want” to obtain the reward. That leaves \(P(K \mid W)\) and \(P(A \mid K, W)\) as the real questions.
For \(P(K \mid W)\), Gowers goes into a lot of detail, but the heuristic is roughly this. Any one AI coming to want something that kills us, directly or by competing with us for resources, may be unlikely. But there are many AI agents, and many wants each of them could have. If there are \(n\) independent ways for it to happen, each with probability \(p\), the probability that at least one of them happens is
\[ 1 - (1 - p)^n, \]
which is close to 1 once \(n p\) is much larger than 1. That is basically the argument of Soares and Yudkowsky that \(P(K \mid W)\) is close to 100%: there are lots of independent wants AI could have, a substantial fraction of them need resources, and we are in the way. Gowers has a few counterarguments, but
The fact that each such argument began with “maybe” or “perhaps” means that at best I can bring my [\(P(K \mid W)\)] temporarily down from high to medium, but I don’t see a way of getting to tiny.
What it means for AI to want, and how others have been trying to pin it down this autumn, deserves a post of its own, which I will write next.