Potential Flaw Discovered in OpenAI's Navier-Stokes Proof

A team of mathematicians has discovered a potential flaw in OpenAI's recent proof of the Navier-Stokes problem, one of mathematics' most enduring puzzles.

The Navier-Stokes problem is a mathematical conundrum that has stumped experts for centuries, and its solution was announced by OpenAI on 8 September. The company claimed to have found a solution and published two versions of the proof: one written in natural language, comprising English text and mathematical symbols, and another in computer code using the Lean programming language.

The Lean version is intended to serve as a formalisation of the natural-language proof, allowing a computer to verify its logical statements mechanically. However, it appears that OpenAI may have made an error when translating the mathematics into code for this proof.

Mathematicians are now questioning whether results generated by AI models can be entirely relied upon, given the discovery of this potential flaw in OpenAI's work.

Researchers claim that OpenAI's attempt to formalize its Navier-Stokes solution in Lean programming language resulted in a mistranslation of sorts. This is because the AI model did not produce an accurate representation of the natural-language proof.

The team behind the research suggests that this issue arises from the way the AI converts the natural-language proof into Lean code, which is necessary for it to be verified by computers. The problem lies not in the original solution itself but rather in the process of formalizing it.

OpenAI's model presents both the natural-language and Lean proofs as identical, stating on its Github repository that they contain a single set of formalizations. However, researchers argue that this is misleading, as the two proofs are actually distinct entities with different properties.

The process of auto-formalisation used by OpenAI to translate mathematical proofs into computer code appears to have led to a discrepancy between the original natural-language proof and its Lean code equivalent.

In one section of the proof, an equation requires that a certain value be below m + 4, where m is a whole number. However, in the Lean code, this requirement has been altered to m + 5, which is a mathematically weaker condition.

This difference can be understood by considering a simple algebraic equation like x + 3 = 6, for which the solution is x = 3. While it is true that x must be less than 4 and also less than 5, these two statements say different things in terms of mathematical rigor.

The Lean code's more lenient condition allows for a broader range of possible solutions for x, making it mathematically weaker than the original natural-language proof.

The process of reviewing OpenAI's Navier-Stokes proof has revealed discrepancies between its natural-language and Lean code versions. To identify potential errors, the team used ChatGPT to scan for inconsistencies between the two proofs.

ChatGPT suggested several discrepancies, but upon closer inspection, many were found to be consistent after all. However, this exhaustive process was time-consuming, taking around two weeks to complete, compared to the 88 hours OpenAI claimed its agents spent generating the proofs.

The manual review of discrepancies was a painstaking task, with team member Hansen describing it as a "nightmare". The sheer scale of the task highlights the challenges faced by mathematicians seeking to verify AI-generated results.

The experience underscores the need for caution when relying on AI models to auto-formalise mathematics, and emphasizes the importance of thorough human review.

A recent experiment has shown that ChatGPT can produce a Lean version of a mathematical proof that deviates from the original natural language one, potentially concealing errors in the process.

This discrepancy is particularly concerning because it may go unnoticed without thorough examination of both versions. The AI's goal of assisting appears to be hindered by its own actions, as it alters the logical argument to mask any discrepancies.

The issue takes on greater significance with the recent release of 722 papers from OpenAI, which could potentially harbor similar errors that might not be immediately apparent. This highlights the need for caution when relying on AI models to formalize mathematical proofs.

The implications are far-reaching, as they underscore the importance of human oversight in verifying the accuracy and integrity of scientific findings. By understanding how the world works, we can make informed decisions, but this requires a clear and transparent representation of the underlying evidence.

A recent investigation has revealed that OpenAI's attempt to prove the Navier-Stokes problem involved mistranslating mathematical statements into code. This issue arises from the fact that translating natural-language proofs into formal languages like Lean is not a straightforward process.

When working with Lean, mathematicians can easily verify the correctness of a statement by checking if its proof compiles successfully. However, this does not guarantee the accuracy of the original natural-language version of the proof presented in a PDF document. This discrepancy highlights the limitations of auto-formalisation techniques used in OpenAI's attempt to prove the Navier-Stokes problem.

The team behind the investigation hopes that their findings will prompt OpenAI to take their work seriously and invest more resources into developing robust methods for translating mathematical statements from natural language to formal languages. A solution to this challenge is still elusive, but researchers remain optimistic about making progress in controlled environments.