www.groundlevel-ai.com/p/what-i-learned-from-the-creators
1 correction found
the first public massive language model training dataset
The Pile was not the first public massive language-model training dataset. Google had already publicly released C4, another massive pretraining corpus for language models, months earlier in 2020.
Full reasoning
This claim is incorrect because a different public, massive language-model pretraining dataset was already available before the Pile.
Evidence:
-
Google publicly introduced C4 on February 24, 2020 as "a new open-source pre-training dataset" and described it as a massive dataset for NLP transfer learning. Google also stated that "C4 is available through TensorFlow Datasets", which shows it was publicly released and accessible before the Pile.
-
The associated T5 paper was originally submitted on October 23, 2019 and says: "By combining the insights from our exploration with scale and our new 'Colossal Clean Crawled Corpus', we achieve state-of-the-art results... To facilitate future work... we release our data set, pre-trained models, and code." That directly contradicts the idea that the Pile was the first public massive language-model training dataset.
-
By contrast, the Pile paper was submitted on December 31, 2020, so it came later.
Because C4 was already a publicly released, large-scale pretraining corpus for language models before December 2020, the Pile cannot accurately be called "the first public massive language model training dataset."
3 sources
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
[Submitted on 31 Dec 2020] ... we present the Pile: an 825 GiB English text corpus targeted at training large-scale language models.
- Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer
February 24, 2020 ... We also introduce a new open-source pre-training dataset, called the Colossal Clean Crawled Corpus (C4) ... To satisfy these requirements, we developed the Colossal Clean Crawled Corpus (C4) ... C4 is available through TensorFlow Datasets.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
[Submitted on 23 Oct 2019] ... By combining the insights from our exploration with scale and our new 'Colossal Clean Crawled Corpus', we achieve state-of-the-art results ... To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.