x.com/spicey_lemonade/status/2045314904014352422
1 correction found
Given Epoch measures fixed-performance inference prices getting ~40x cheaper per year (90% CI: 10x-900x).
This misstates Epoch AI's result. Epoch gives 40x/year as one example, while its overall finding is that fixed-performance inference price declines vary widely across benchmarks, ranging from about 9x to 900x per year—not a 90% confidence interval around 40x.
Full reasoning
Epoch AI's writeup does not say that fixed-performance inference prices are generally getting "~40x cheaper per year" with a "90% CI" of 10x–900x.
What the source actually says is:
- "For instance, the price to achieve GPT-4's performance on a set of PhD-level science questions fell by 40x per year." That makes 40x/year an example for one specific performance threshold, not Epoch's single overall estimate.
- The same page then says "The rate of decline varies dramatically depending on the performance milestone, ranging from 9x to 900x per year." That is a range across different milestones/benchmarks, not a confidence interval around 40x.
So the post compresses two different ideas into one incorrect statistical claim: it turns a single illustrative 40x example into the general estimate, and it relabels Epoch's cross-benchmark range as a "90% CI."
1 source
- LLM inference prices have fallen rapidly but unequally across tasks | Epoch AI
For instance, the price to achieve GPT-4's performance on a set of PhD-level science questions fell by 40x per year. The rate of decline varies dramatically depending on the performance milestone, ranging from 9x to 900x per year.