www.lesswrong.com/posts/45oxYwysFiqwfKCcN/five-hinge-questions-that-decide-wheth...
1 correction found
Six doublings in frontier training compute between GPT‑2 (2019) and GPT‑4 (2023)
This understates the compute increase. Epoch AI places GPT‑2 below 10^23 FLOP and GPT‑4 at about 2.1×10^25 FLOP, which implies more than seven doublings even under the most conservative reading.
Full reasoning
Epoch AI’s 2024 large-model survey states that “GPT and GPT-2 were below 10^23 FLOP”. Its later GPT-4-scale survey estimates GPT-4 at 2.1e25 FLOP.
Those two figures cannot be reconciled with “six doublings”:
- If GPT‑2 were as high as the maximum allowed by the first source (just under 10^23 FLOP), then GPT‑4 would still be more than 210× larger in training compute.
- A 210× increase is more than 7 doublings (since 2^7 = 128 and 2^8 = 256).
- Because GPT‑2 is described as below 10^23 FLOP, the real increase is larger than that lower-bound calculation.
So the post’s “six doublings” claim is not just imprecise; it is materially too low relative to the cited compute estimates for GPT‑2 and GPT‑4.
2 sources
- Tracking large-scale AI models | 81 models across 18 countries | Epoch AI
Figure 4 shows how the next model trained at a similar scale, Google's GPT-inspired Meena, was not published until 2020. GPT and GPT-2 were below 10^23 FLOP, but GPT-3 arrived later the same year...
- Over 30 AI models have been trained at the scale of GPT-4 | Epoch AI
Models included in this dataset: ... GPT-4 2.1e25 Low-precision Compute estimated using training hardware and training duration.