x.com/MajmudarAdam/status/2098881885200081234
1 correction found
a complete inability to do advanced math before the o-series
This overstates earlier models’ weaknesses. Before the o-series, OpenAI had already reported models solving nontrivial competition-math problems, including AIME/IMO-style problems, and GPT-4o solving some AIME questions.
Full reasoning
Saying pre-o-series models had a "complete inability" to do advanced math is too strong and is contradicted by OpenAI's own published results.
- In OpenAI’s September 12, 2024 writeup on o1, the company explicitly compared GPT-4o (which predates the o-series) on the 2024 AIME exams and reported that "GPT‑4o only solved on average 12% (1.8/15) of problems." That is much worse than o1, but it is not a complete inability to do advanced math.
- Earlier still, OpenAI’s February 2, 2022 post "Solving (some) formal math olympiad problems" reported a language-model-based theorem prover that solved challenging high-school olympiad problems, including AIME problems and two problems adapted from the IMO.
So the broad point that math capability improved a lot may be fair, but the specific claim that models had a complete inability before the o-series is factually incorrect.
2 sources
- Learning to reason with LLMs | OpenAI
On the 2024 AIME exams, GPT‑4o only solved on average 12% (1.8/15) of problems.
- Solving (some) formal math olympiad problems | OpenAI
We built a neural theorem prover for Lean that learned to solve a variety of challenging high-school olympiad problems, including problems from the AMC12 and AIME competitions, as well as two problems adapted from the IMO.