www.lesswrong.com/posts/9gqjdRyJxSM6kS7zi/i-trained-gpt-2-with-full-batch-gradie...
1 correction found
the full gradient and update at every step of the full-batch run
The uploaded per-step gradients and updates are not actually full. The dataset card says they omit the 155M-parameter `value_embeds` tensor to save space.
Full reasoning
The linked Hugging Face dataset card explicitly says the saved per-step gradients and updates are incomplete:
- "Gradients and updates leave out value_embeds (155M parameters) to save space; the checkpoints include it."
So the Hugging Face upload does not contain the full gradient and full update at every step. It contains per-step gradients/updates for most parameters, but excludes one large parameter group (value_embeds). Readers who click through expecting complete per-step tensors would be misled by the word "full" here.
2 sources
- shawnghu/nanogpt-fullbatch · Datasets at Hugging Face
Gradients and updates leave out value_embeds (155M parameters) to save space; the checkpoints include it.
- I trained GPT-2 with full-batch gradient descent so you don't have to - LessWrong
Everything the runs produced is on Hugging Face: metrics and dynamics statistics for every run, the full gradient and update at every step of the full-batch run...