x.com/gabriberton/status/2051873677998956851
1 correction found
When they trained GPT3 they had loss spikes because they scraped from a subreddit of microwave noises
The "microwavegang" loss-spike anecdote was not about GPT-3. Nathan Lambert said it was an Ai2 example that affected "our earlier models," not OpenAI's GPT-3.
Full reasoning
The claim appears to misattribute a real anecdote to the wrong model.
The source of the "microwavegang" story is a 2025 Lex Fridman podcast transcript. In that transcript, Nathan Lambert explicitly says: "Can I give Ai2’s example of what blew up our earlier models is a Subreddit called microwavegang" and then explains that the subreddit contains long runs of the letter "M," which caused high-loss batches for Ai2's earlier models. He immediately adds: "This is old, this is not recent."
That is not the same as saying this happened while training GPT-3. GPT-3 is OpenAI's model, described in the original paper Language Models are Few-Shot Learners, whose authors are from OpenAI. So the viral anecdote is a real training-data story, but the post incorrectly assigns it to GPT-3/OpenAI rather than Ai2's older models.
In short: "microwavegang caused loss spikes" is an Ai2 anecdote; "when they trained GPT3" is the incorrect part.
2 sources
- Transcript for DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459 - Lex Fridman
Nathan Lambert says: "Can I give Ai2’s example of what blew up our earlier models is a Subreddit called microwavegang..." and later: "This is old, this is not recent."
- [2005.14165] Language Models are Few-Shot Learners
The original GPT-3 paper states: "Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters" and lists OpenAI authors, showing GPT-3 is OpenAI's model, not Ai2's earlier models.