www.dwarkesh.com/p/bits-per-sample
2 corrections found
If it took you X many FLOPs to go from 1/100,000 pass rate to 1/10,000, then it will take you X many FLOPs to go from 1/10,000 pass rate to 1/1,000.
A power-law scaling relationship does not imply equal absolute FLOP increments for each 10× improvement. It implies a constant multiplicative factor in compute, so the later 10× gain generally costs more additional FLOPs than the earlier one.
Full reasoning
The cited "power law" premise does not support this conclusion.
Kaplan et al.'s scaling-laws paper says language-model loss scales as a power law with compute. And a power law means a multiplicative/relative relationship between variables, not a claim that equal improvements require equal absolute additions of compute.
Mathematically, if pass rate (or any proxy performance measure) followed a power law in compute,
[
p = a C^b
]
then increasing pass rate by a factor of 10 requires multiplying compute by a constant factor:
[
\frac{C_2}{C_1} = 10^{1/b}.
]
That means the ratio of compute is constant across decades, but the extra FLOPs are not:
[
C_2 - C_1 = (10^{1/b}-1)C_1.
]
Because that difference is proportional to the starting compute level (C_1), the later decade of improvement costs more additional FLOPs than the earlier one. So going from 1/10,000 to 1/1,000 would not, in general, take the same number of extra FLOPs as going from 1/100,000 to 1/10,000.
So the statement confuses equal multiplicative scaling with equal additive compute increments. A power law would justify equal spacing on a log-compute axis (or constant compute ratios), not the claim that each 10× pass-rate gain consumes the same absolute FLOP budget.
2 sources
- Scaling Laws for Neural Language Models
Abstract: We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training...
- power law - Mathematics - Loterre
Definition: In statistics, a power law is a functional relationship between two quantities, where a relative change in one quantity results in a relative change in the other quantity proportional to a power of the change... The rate of change exhibited in these relationships is said to be multiplicative.
So, we should actually chart the pass rate on a log scale - again, to account for how each increment in the x-axis corresponds to the same number of FLOPs.
A log scale does not make equal x-axis steps correspond to the same absolute number of FLOPs under a power law. It makes them correspond to equal multiplicative changes, not equal additive compute budgets.
Full reasoning
This sentence makes the same mathematical mistake as the previous one.
A power-law relationship means performance changes with multiplicative changes in compute. If pass rate scaled like (p = aC^b), then equal intervals in log(pass rate) correspond to equal intervals in log(compute)—that is, equal multiplicative factors of compute.
They do not correspond to the same absolute number of FLOPs. For example, multiplying compute by 10 from 1e20 to 1e21 FLOPs adds 9e20 FLOPs, while multiplying by 10 again from 1e21 to 1e22 adds 9e21 FLOPs—ten times more additional compute.
So using a log axis can be a sensible way to visualize multiplicative scaling, but it does not mean each x-axis increment represents the same number of FLOPs. It represents the same factor change in FLOPs.
2 sources
- Scaling Laws for Neural Language Models
Abstract: We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training...
- power law - Mathematics - Loterre
Definition: In statistics, a power law is a functional relationship between two quantities, where a relative change in one quantity results in a relative change in the other quantity proportional to a power of the change... The rate of change exhibited in these relationships is said to be multiplicative.