www.astralcodexten.com/p/nostalgebraists-hydrogen-jukeboxes
1 correction found
R1 is a small model - certainly today, but even by the standards of early 2025 when it was trained.
DeepSeek-R1 was not a small model. DeepSeek’s own materials list R1 at 671B total parameters with 37B activated per token, and distinguish it from separate smaller distilled models.
Full reasoning
This appears to confuse DeepSeek-R1 itself with its distilled smaller variants.
DeepSeek's official README for R1 lists DeepSeek-R1 as a model with 671B total parameters and 37B activated parameters, built on DeepSeek-V3-Base. The same README separately lists the genuinely smaller models as DeepSeek-R1-Distill checkpoints at 1.5B, 7B, 8B, 14B, 32B, and 70B.
DeepSeek's official V3 announcement likewise describes the underlying V3 base as a 671B MoE model with 37B activated parameters. So the published documentation does not support calling R1 itself "a small model"; rather, it was a frontier-scale MoE model, with smaller distilled descendants released alongside it.
Because the official sources explicitly separate R1 from the small distilled models, describing R1 itself as small is inaccurate.
2 sources
- DeepSeek-R1 README
DeepSeek-R1 Models ... DeepSeek-R1 | 671B | 37B | 128K ... DeepSeek-R1-Distill-Qwen-1.5B ... 7B ... 8B ... 14B ... 32B ... 70B
- Introducing DeepSeek-V3
What's new in V3: 671B MoE parameters, 37B activated parameters, Trained on 14.8T high-quality tokens