Lack of trust in AI-generated proofs and the reliability of proof checkers

Signals over time

2 observed signals across 1 sources, tracked for 4 days. Confidence: low.

What people are saying Observed

• “> If AI advances more and becomes smarter it can be used on both sides. Yes, and that's true even if it doesn't advance or become smarter. It's happening already that people are using AI on "both sides" both adversarially and defensively and that is improving software, and hopefully mathematics as well. > I am not sure about the Navier-Stokes proof but in general it sounds highly unlikely that many well-known theorems turn out false. I'm talking about the LLM Navier-Stokes proof specifically because it's very long and complicated and an LLM is not the same as a human mathematician. A human mathematician is trying to use the tools in a truth-seeking way, whereas LLMs are trying to satisfy the goal of presenting a proof that convinces people it is correct. Maybe naively, it seems to me the failure modes of proof assistants would be amplified under adversarial conditions (i.e. driven by LLMs). Completely agree that there seem to be many ways that relying on proof assistant software can go wrong.”

  Hacker News · frustration

• “Will we ever see a soundness bug in the lean kernel again? To a software developer the question seems insane. There were bugs in the past, of course there will be more. When we see those bugs, what will it mean for AI lean proofs? Can we trust them? This whole thing boils down to trust. We can trust human verifications highly because of community and reputation and human proof-of-work. Humans sometimes lie about math results but it's rare because of this. They make mistakes and those mistakes are discovered by communities who are themselves largely trustworthy because of this. LLMs don't care about reputation. They hallucinate and fabricate often. In harnesses they literally try to cheat and bend rules, which is a disaster for knowledge that is encoded in rules. So we have to rely on proof checkers. The problem is, how trustworthy are proof checkers? Are there bugs? And is the underlying theory itself free of paradoxes and unknowables and mathematical "bugs" that can be exploited? Imagine a human mathematician hell bent on deceiving other mathematicians - would we trust their breakthroughs, even with a verifier? Because of these properties, the final backstop ha”

  Hacker News · frustration