The technical debt of AI-generated mathematics
Cohn reaches for technical debt as the analogy, then finds it too kind: in software engineering a program with technical debt still runs, while in mathematics getting the answer is barely a form of working at all. I’d add the part open source already knows. Technical debt is only a loan when the people creating it are the people who pay it back. Otherwise it is work moved onto somebody else.
Mochizuki’s purported proof of the ABC conjecture is an instructive example. He’s a brilliant mathematician, and I haven’t studied his work myself, so let’s give it the benefit of the doubt and assume he really has thought through a valid proof. That would be a magnificent individual achievement, but the broader mathematical community has not been able to fully understand his papers on this topic, he has not been able to bridge this gap in understanding, and as a result his work has not had the impact one would hope for.
Mochizuki is the best human-generated example of a proof shaped like the AI-generated ones: granted valid, and still not absorbed by anyone. So the failure mode isn’t new. What is new is how quickly we can produce it. OpenAI reported a result 88 hours after launching the search for the Navier–Stokes proof, and the next search can start tomorrow.
The net result is that everyone benefits, in a way that transcends competition. For example, Maryna Viazovska won the Fields Medal by solving a problem I had tried to solve for many years. […] her solution greatly enriched human understanding, and it opened up many lines of work I could contribute to. It was obviously a net win for everyone, including me.
[…] However, the primary value of a solution lies in the insight and understanding it provides, not in the mere fact that the problem was solved.
That net win is what mathematicians are looking for from a proof, and so far it isn’t there in the AI-assisted ones. Viazovska’s solution opened up lines of work for the person she beat to it. A proof nobody can read doesn’t open up anything.
AI often produces cryptic, messy, ad hoc, poorly motivated solutions full of elaborate calculations and unnecessary complications. Basically, they often produce math slop. It may be correct, and it may contain genuinely novel and important ideas, but the process of going through it carefully can be awful. […] By comparison, human work often goes through a slop stage early on, but we try hard not to leave it at that.
[…] I just don’t enjoy reading poorly explained mathematics, even if with considerable effort I can extract something much more appealing, and I’m not the only one.
Again, much the same as reviewing an AI-generated PR. Our own first drafts are slop too; the difference is that we clean them up before asking anybody to read them. And “it passes CI” doesn’t answer that.
Mathematical truth may be easier to train for than effective communication with humans, which is more subtle and less objective.
This is the reward problem from my other post. Correctness has verifiable signals to train on, and understanding doesn’t. Communication doesn’t have one either, which is why models end up much better at getting the answer right than at explaining it.
This communication issue is why many mathematicians are unhappy when someone has AI solve an open problem, dumps a slop proof on the internet, and then abandons it, without any attempt to understand it themselves or communicate it effectively to others. They believe they are making a contribution, but often they are primarily creating unpleasant work for other people.
Closer than my own zero-day comparison, which picks up a different harm. From the open source maintainer’s perspective, dropping a feature does no apparent harm; the harm comes only if you merge it. In the bug disclosure analogy, the mere act of dropping the facts does the harm, as that post argues.
He compares it closer to what happens to open source maintainers, and that is the better fit here. It is the drive-by AI pull request: someone opens a PR for a feature they don’t understand, in a codebase they haven’t read, and the work of making it safe to merge lands on people who never asked for the feature. They usually mean well, which is what makes it awkward to refuse.
Math slop causes technical debt in mathematics, and in some ways it’s even more insidious. In software engineering, a functioning program with technical debt still works, and deliberately accepting technical debt may be justified if you truly need it to work immediately. In mathematics, my experience is that getting the answer is usually a negligible form of “working.”
So Cohn’s view is that something here is special to mathematics: the answer on its own is worth almost nothing. I’d argue mathematics isn’t alone in this, and that it sits at one end of a spectrum software is already spread across. Sean Goedecke splits programming into two kinds: small codebases with low turnover, where you are expected to hold the whole thing in your head, and large codebases with high turnover, where a partial and somewhat wrong theory is the best anyone gets. His point is that the second kind is legitimate, and I agree. Mine is that mathematics is the first kind all the way down, and so is the research software that establishes a scientific result, which is roughly what I argued about correctness and V&V. Goedecke also argues that a theory of the codebase is one value among many, traded off against speed, or legal compliance, or politics, and he is right that we make that trade every day. That works when what you want from the software is that it runs. When what you want is that it is true, the understanding is part of the product, so trading it away is trading away what you came for.
Goedecke writes explicitly against Naur’s “programming as theory building”, which is the frame I used for dead programs. I don’t think we disagree as much as it looks: he is writing about software where the product is the execution, and I was writing about software where the product is the theory.
To a certain extent technical debt is unavoidable: nobody has the wisdom or foresight to avoid it completely. However, in the past it was kept in check socially, since the people creating the debt were largely the same as those who would have to pay it in the future. […] In contrast, it’s a problem if people have a button that creates technical debt without their knowledge.
We live in a world with an infinite supply of slop: anyone can type a prompt and produce output on whatever topic they choose. Pushing the AI button is generally not a significant contribution, since we all have buttons and can push them as we see fit. The contribution comes from what you do with the output.
Again this is open source maintenance, and the key question is who is doing it to whom. Within one community the debt stays in check because the people creating it expect to pay it: you can rush a change through a codebase you will still be maintaining next year, and the bill comes back to you. When outsiders push the button into somebody else’s field, that loop is cut. Same community, or one dumping on another?
Instead, you should take responsibility for your contributions and work in good faith to help integrate them successfully into human knowledge and understanding. […] There is no universal solution, but the key point is that if this project was worth doing and publicizing in the first place, then it’s worth doing responsibly.
Which is where responsible disclosure ended up too. Publishing and saying nothing are not the only two options, and the third one costs the finder some work.
The parallel with unpaid open source maintenance leads somewhere Cohn doesn’t go, and it is about money. If the answer is the cheap part and integrating it is the expensive part, then whoever publicizes the answer should be funding the rest. We already say it is irresponsible for tech companies to drop features into a project and expect volunteers to maintain them without paying for the development. I’d say it is irresponsible in the same way to drop a 165-page proof into a field and expect the field to absorb it. Cohn asks individuals to take responsibility for their contributions, which is right. But the loudest button-pushers are companies, and asking a company to take responsibility usually means asking it to pay.